加快时间戳操作

Question

The following transformation (ms -> datetime -> conver timezone) takes a long time to run (4 minutes), probably because I am working with a large dataframe:

for column in ['A', 'B', 'C', 'D', 'E']:
    # Data comes in unix time (ms) so I need to convert it to datetime
    df[column] = pd.to_datetime(df[column], unit='ms')

    # Get times in EST
    df[column] = df[column].apply(lambda x: x.tz_localize('UTC').tz_convert('US/Eastern'))

Is there any way to speed it up? Am I already using Pandas data structures and methods in the most efficient manner?

Answer 1

These are available as DatetimeIndex methods which will be much faster:

df[column] = pd.DatetimeIndex(df[column]).tz_localize('UTC').tz_convert('US/Eastern')

Note: In 0.15.0 you'll have access to these as Series dt accessor :

df[column] = df[column].dt.tz_localize('UTC').tz_convert('US/Eastern')

Answer 2

I would give this a try in Bash using the date command. Date proves to be faster than even gawk for routine conversions. Python may struggle with this.

To speed this up even faster export the column in A in one temp file, column B in another, ect. (You can even do this in python). Then run the 5 columns in parallel.

for column in ['A']:
  print>>thefileA, column
for column in ['B']:
  print>>thefileB, column

Then a bash script:

#!/usr/bin/env bash
readarray a < thefileA
for i in $( a ); do
    date -r item: $i
done

You'll want a master bash script to run the first part in python python pythonscript.py . Then you will want to call in each of the bash scripts in the background from the master ./FILEA.sh & . That will run each column individually and will automatically designate nodes. For my bash loop after readarray, I'm not 100% that is the correct syntax. If you are on a linux, use date -d @ item .