Python convert a series of seconds since epoch time to datetime series

Question

Is there any fast way to convert a series of seconds since epoch time to datetime object series?

I use:

for i in range(len(df)):
    df['datetime'].iloc[i] = datetime.fromtimestamp(df['epochtime'].iloc[i])

But it is very slow since my dataframe is very large. Is there any fast way to do this? like pandas function?

Answer 1

you can use to_datetime(..., unit='s') :

df['datetime'] = pd.to_datetime(df['epochtime'], unit='s')

Time comparison:

In [158]: df = pd.DataFrame({'epochtime': pd.date_range('2001-01-01', freq='1S', periods=10**5)}).astype(np.int64)//10**9

In [159]:

In [159]: df.head()
Out[159]:
   epochtime
0  978307200
1  978307201
2  978307202
3  978307203
4  978307204

In [160]:

In [160]: len(df)
Out[160]: 100000

In [161]:

In [161]: %timeit df['datetime'] = pd.to_datetime(df['epochtime'], unit='s')
100 loops, best of 3: 16.9 ms per loop

In [162]:

In [162]: %%timeit
   .....: for i in range(len(df)):
   .....:     df['datetime'].iloc[i] = datetime.fromtimestamp(df2['epochtime'].iloc[i])
   .....:
c:\envs\py35\lib\site-packages\pandas\core\indexing.py:128: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)
1 loop, best of 3: 54.5 s per loop

Conclusion: @Natecat, as you can see 16.9 ms vs. 54.5 s