Fast conversion of time string (Hour:Min:Sec.Millsecs) to float

Question

I use pandas to import a csv file (about a million rows, 5 columns) that contains one column of timestamps (increasing row-by-row) in the format Hour:Min:Sec.Millsecs, eg

11:52:55.162

and some other columns with floats. I need to transform the timestamp column into floats (say in seconds). So far I'm using

pandas.read_csv  

to get a dataframe df and then transform it into a numpy array

df=np.array(df)

All the above works great and is quite fast. However, then I use datetime.strptime (the 0th columns are the timestamps)

df[:,0]=[(datetime.strptime(str(d),'%H:%M:%S.%f')).total_seconds() for d in df[:,0]]

to transform the timestamps into seconds and unfortunately this turns out to be veryyyy slow . It's not the iteration over all the rows that so slow but

datetime.strptime 

is the bottleneck. Is there a better way to do it?

Answer 1

Here, using timedeltas

Create a sample series

In [21]: s = pd.to_timedelta(np.arange(100000),unit='s')

In [22]: s
Out[22]: 
0    00:00:00
1    00:00:01
2    00:00:02
3    00:00:03
4    00:00:04
5    00:00:05
6    00:00:06
7    00:00:07
8    00:00:08
9    00:00:09
10   00:00:10
11   00:00:11
12   00:00:12
13   00:00:13
14   00:00:14
...
99985   1 days, 03:46:25
99986   1 days, 03:46:26
99987   1 days, 03:46:27
99988   1 days, 03:46:28
99989   1 days, 03:46:29
99990   1 days, 03:46:30
99991   1 days, 03:46:31
99992   1 days, 03:46:32
99993   1 days, 03:46:33
99994   1 days, 03:46:34
99995   1 days, 03:46:35
99996   1 days, 03:46:36
99997   1 days, 03:46:37
99998   1 days, 03:46:38
99999   1 days, 03:46:39
Length: 100000, dtype: timedelta64[ns]

Convert to string for testing purposes

In [23]: t = s.apply(pd.tslib.repr_timedelta64)

These are strings

In [24]: t.iloc[-1]
Out[24]: '1 days, 03:46:39'

Dividing by a timedelta64 converts this to seconds

In [25]: pd.to_timedelta(t.iloc[-1])/np.timedelta64(1,'s')
Out[25]: 99999.0

This is currently matching using a reg-ex, so not very fast from a string directly.

In [27]: %timeit pd.to_timedelta(t)/np.timedelta64(1,'s')
1 loops, best of 3: 1.84 s per loop

This is a date-timestamp based soln

Since date times are already stored as int64's this is very easy an fast

Create a sample series

In [7]: s = Series(date_range('20130101',periods=1000,freq='ms'))

In [8]: s
Out[8]: 
0           2013-01-01 00:00:00
1    2013-01-01 00:00:00.001000
2    2013-01-01 00:00:00.002000
3    2013-01-01 00:00:00.003000
4    2013-01-01 00:00:00.004000
5    2013-01-01 00:00:00.005000
6    2013-01-01 00:00:00.006000
7    2013-01-01 00:00:00.007000
8    2013-01-01 00:00:00.008000
9    2013-01-01 00:00:00.009000
10   2013-01-01 00:00:00.010000
11   2013-01-01 00:00:00.011000
12   2013-01-01 00:00:00.012000
13   2013-01-01 00:00:00.013000
14   2013-01-01 00:00:00.014000
...
985   2013-01-01 00:00:00.985000
986   2013-01-01 00:00:00.986000
987   2013-01-01 00:00:00.987000
988   2013-01-01 00:00:00.988000
989   2013-01-01 00:00:00.989000
990   2013-01-01 00:00:00.990000
991   2013-01-01 00:00:00.991000
992   2013-01-01 00:00:00.992000
993   2013-01-01 00:00:00.993000
994   2013-01-01 00:00:00.994000
995   2013-01-01 00:00:00.995000
996   2013-01-01 00:00:00.996000
997   2013-01-01 00:00:00.997000
998   2013-01-01 00:00:00.998000
999   2013-01-01 00:00:00.999000
Length: 1000, dtype: datetime64[ns]

Convert to ns since epoch / divide to get ms since epoch (if you want seconds, divide by 10**9)

In [9]: pd.DatetimeIndex(s).asi8/10**6
Out[9]: 
array([1356998400000, 1356998400001, 1356998400002, 1356998400003,
       1356998400004, 1356998400005, 1356998400006, 1356998400007,
       1356998400008, 1356998400009, 1356998400010, 1356998400011,
       ...
       1356998400992, 1356998400993, 1356998400994, 1356998400995,
       1356998400996, 1356998400997, 1356998400998, 1356998400999])

Pretty fast

In [12]: s = Series(date_range('20130101',periods=1000000,freq='ms'))

In [13]: %timeit pd.DatetimeIndex(s).asi8/10**6
100 loops, best of 3: 11 ms per loop

Answer 2

I'm guessing that the datetime object has a lot of overhead - it may be easier to do it by hand:

def to_seconds(s):
    hr, min, sec = [float(x) for x in s.split(':')]
    return hr*3600 + min*60 + sec

Answer 3

Using sum() , and enumerate() -

>>> ts = '11:52:55.162'
>>> ts1 = map(float, ts.split(':'))
>>> ts1
[11.0, 52.0, 55.162]
>>> ts2 = [60**(2-i)*n for i, n in enumerate(ts1)]
>>> ts2
[39600.0, 3120.0, 55.162]
>>> ts3 = sum(ts2)
>>> ts3
42775.162
>>> seconds = sum(60**(2-i)*n for i, n in enumerate(map(float, ts.split(':'))))
>>> seconds
42775.162
>>>