使用apply方法提高熊猫的表现

Question

I'm working on pandas for high performance calculations, the below function gives 1 loop, best of 5: 7.24 s per loop for 50,000 rows.

I have to scale it to 1 million rows.

How to vectorise the function and apply to all rows. So that overall performance can be improved?

def weightedFlowAmt(startDate,endDate,tradeDate,tradeAmt):
  startInDays = datetime.strptime(startDate, "%Y-%m-%d")
  endInDays = datetime.strptime(endDate, "%Y-%m-%d")
  tradeInDays = datetime.strptime(tradeDate, "%Y-%m-%d")
  differenceTradeAndEnd=abs((endInDays - tradeInDays).days)
  differenceStartAndEnd=abs((endInDays - startInDays).days)
  weighted_FlowAmt = (tradeAmt * differenceTradeAndEnd)/differenceStartAndEnd

mutatedCashFlow['flow'] = mutatedCashFlow.apply(lambda row:
        weightedFlowAmt(row['startDate'], row['EndDate'], row['tradeDate'],
                        row['tradeAmount']),
    axis=1)

Answer 1

I think you can remove apply and use vectorized functions:

mutatedCashFlow['startDate'] = pd.to_datetime(mutatedCashFlow['startDate'])
mutatedCashFlow['EndDate'] = pd.to_datetime(mutatedCashFlow['EndDate'])
mutatedCashFlow['tradeDate'] = pd.to_datetime(mutatedCashFlow['tradeDate'])

diffTradeAndEnd=((mutatedCashFlow['EndDate']-mutatedCashFlow['tradeDate']).dt.days).abs()
diffStartAndEnd=((mutatedCashFlow['EndDate']-mutatedCashFlow['startDate']).dt.days).abs()

mutatedCashFlow['flow'] = (mutatedCashFlow['tradeAmount']*diffTradeAndEnd)/diffStartAndEnd

Alternative:

mutatedCashFlow['startDate'] = pd.to_datetime(mutatedCashFlow['startDate'])
mutatedCashFlow['EndDate'] = pd.to_datetime(mutatedCashFlow['EndDate'])
mutatedCashFlow['tradeDate'] = pd.to_datetime(mutatedCashFlow['tradeDate'])

diffTradeAndEnd=mutatedCashFlow['EndDate'].sub(mutatedCashFlow['tradeDate']).dt.days.abs()
diffStartAndEnd=mutatedCashFlow['EndDate'].sub(mutatedCashFlow['startDate']).dt.days.abs()

mutatedCashFlow['flow'] = mutatedCashFlow['tradeAmount'].mul(diffTradeAndEnd)
                                                        .div(diffStartAndEnd)
print (mutatedCashFlow)