Convert timestamp ISO to datetime in Python?

Question

I am trying to convert an ISO timestamp column in my dataset to datetime. I'm able to successfully convert certain rows but others fail without a noticeable pattern.

Here is what my raw data looks like:

911     2015-10-15T12:39:36
2520    2015-10-02T14:54:58
2545    2015-09-18T21:07:40
805     2015-10-28T17:17:22

I try to run this code on it:

datetime.strptime(orders['Timestamp'][58], "%Y-%m-%dT%H:%M:%S")

Sometimes it works and turns into datetime:

2015-05-16 08:46:10
2015-05-15 17:02:04
2015-05-15 16:43:42
2015-05-15 16:40:16

Every 50 rows or so it throws up an error:

KeyError                                  Traceback (most recent call last)
<ipython-input-130-2db5a7ab5914> in <module>()
      1 for i in range(116, len(orders['Timestamp'])):
----> 2     df_dt=datetime.strptime(orders['Timestamp'][i],"%Y-%m-%dT%H:%M:%S")
      3     print df_dt

c:\python27\lib\site-packages\pandas\core\series.pyc in __getitem__(self, key)
    549     def __getitem__(self, key):
    550         try:
--> 551             result = self.index.get_value(self, key)
    552 
    553             if not np.isscalar(result):

c:\python27\lib\site-packages\pandas\core\index.pyc in get_value(self, series, key)
   1721 
   1722         try:
-> 1723             return self._engine.get_value(s, k)
   1724         except KeyError as e1:
   1725             if len(self) > 0 and self.inferred_type in ['integer','boolean']:

pandas\index.pyx in pandas.index.IndexEngine.get_value (pandas\index.c:3204)()

pandas\index.pyx in pandas.index.IndexEngine.get_value (pandas\index.c:2903)()

pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:3843)()

pandas\hashtable.pyx in pandas.hashtable.Int64HashTable.get_item (pandas\hashtable.c:6525)()

pandas\hashtable.pyx in pandas.hashtable.Int64HashTable.get_item (pandas\hashtable.c:6463)()

KeyError: 268L

Can't tell why the other dates are successfully converted but not these (can't see a pattern):

2015-05-30T22:25:52
2015-03-04T03:57:51
2013-11-22T22:28:23

Thanks.

Answer 1

You can use to_datetime :

In [11]: df
Out[11]:
      0                    1
0   911  2015-10-15T12:39:36
1  2520  2015-10-02T14:54:58
2  2545  2015-09-18T21:07:40
3   805  2015-10-28T17:17:22

In [12]: pd.to_datetime(df[1])
Out[12]:
0   2015-10-15 12:39:36
1   2015-10-02 14:54:58
2   2015-09-18 21:07:40
3   2015-10-28 17:17:22
Name: 1, dtype: datetime64[ns]

Answer 2

The error your script is throwing is KeyError , not ValueError (which would have been the case if your date format had any problems). Make sure that the key is there in the dicts by something like this before using them:

EDIT After having a second look into your error log, I realized that the KeyError isn't triggered in your script, but the core pandas module: c:\\python27\\lib\\site-packages\\pandas\\core\\index.pyc . So, as I explained in the comments, this is a pandas core issue. You may raise a bug on their issue tracker by mentioning these details.

Answer 3

If your ISO is in the proper format (not str), you can just:

from datetime import datetime

datetime.timestamp(timestamp_ISO_8601)

This will turn your timestamp to Epoch. From there, you can convert to anything you want. That's how I do it.