做这些数据框操作的更快方法？

Question

I am loading a dataframe from csv, and then performing the operations below. Loading the dataframe takes about 2 seconds. The other operations ( mainly the date conversions ) take 30 seconds. Is there a way to speed up the other operations?

df = pd.DataFrame.from_csv( fn, index_col=None )

df['SCHEDULED_OFF'] = pd.to_datetime( df['SCHEDULED_OFF'], format='%d-%m-%Y %H:%M' )
df['LATEST_TAKEN'] = pd.to_datetime( df['LATEST_TAKEN'], format='%d-%m-%Y %H:%M:%S' )
df['FIRST_TAKEN'] = pd.to_datetime( df['FIRST_TAKEN'], format='%d-%m-%Y %H:%M:%S' )
df['SETTLED_DATE'] = pd.to_datetime( df['SETTLED_DATE'], format='%d-%m-%Y %H:%M:%S' )
df['ACTUAL_OFF'] = pd.to_datetime( df['ACTUAL_OFF'], format='%d-%m-%Y %H:%M:%S' )
df['ACTUAL_OFF'] = df['ACTUAL_OFF'].fillna( pd.datetime.min )
df[ 'LATEST_TAKEN_FROM_SCHEDULED_OFF' ] = ( df['SCHEDULED_OFF'].values -df['LATEST_TAKEN'].values ) / np.timedelta64( 1, 's' )
df[ 'FIRST_TAKEN_FROM_SCHEDULED_OFF' ] = ( df['SCHEDULED_OFF'].values -df['FIRST_TAKEN'].values ) / np.timedelta64( 1, 's' )
df[ 'IN_PLAY' ] = [ dicInPlay[ x ] for x in df[ 'IN_PLAY' ] ]
df['COUNTRY'] = df['COUNTRY'].fillna( '' )
df['FULL_DESCRIPTION'] = df['FULL_DESCRIPTION'].fillna( '' )
df['EVENT'] = df['EVENT'].fillna( '' )
df['COURSE'] = df['COURSE'].fillna( '' )

Answer 1

Not really a solution, but a way to do this faster is having the dates in standard ISO format ...

To illustrate this can make a big difference, some timings (with a column of 10000 date strings):

# with standard ISO formatted strings (%Y-%m-%d %H:%M:%S)
In [52]: %timeit pd.to_datetime(df['date'])
100 loops, best of 3: 2.88 ms per loop

# with your dayfirst-like format (%d-%m-%Y %H:%M)
In [66]: %timeit pd.to_datetime(df['date'], format='%d-%m-%Y %H:%M')
10 loops, best of 3: 78.2 ms per loop

In [67]: %timeit pd.to_datetime(df['date'], dayfirst=True)
1 loops, best of 3: 800 ms per loop

So I think part of the reason it is slow, is this date parsing (20-30 time slowdown when not having standard ISO format). And I don't know if this can be further enhanced if you can't change the format.

For the other lines I don't directly see a possible spead-up, only for [ dicInPlay[ x ] for x in df[ 'IN_PLAY' ] ] you could test if df['IN_PLAY'].map(dicInPlay) is faster.