将带有时间戳字符串的Pandas DF转换为不带tz的本地日期时间

Question

I have DataFrame column with string timestamp and I want to convert it to local datetime without time zone.

df = {'id' : [1, 2, 3],
      'timestamp' : ['2019-07-01T21:30:20Z', '2019-07-02T21:30:20Z', '2019-07-03T21:30:20Z']}

df = pd.DataFrame(df, columns = ['id','timestamp'])

My code:

df['timestamp'] = (pd.DatetimeIndex(pd.to_datetime(df['timestamp'], format="%Y-%m-%dT%H:%M:%SZ",
  errors='ignore')).tz_localize('UTC').tz_convert('Europe/Prague'))

This works for example above but not for example below. OutOfBoundsDatetime: Out of bounds nanosecond timestamp

df = {'id' : [1, 2, 3],
      'timestamp' : ['2019-07-01T21:30:20Z', '2999-12-31T21:30:20Z', '9999-12-30T21:30:20Z']}

It is problem with 2999-12-31T21:30:20Z or 9999-12-30T21:30:20Z. How can I solve it?

Answer 1

So the max timestamp pandas can hold is '2262-04-11 23:47:16.854775807' (see https://pandas-docs.github.io/pandas-docs-travis/user_guide/timeseries.html#timestamp-limitations ). Therefore both the second and third timestamps are the issues.

As for the fix if you need those dates I am unsure if I can help you. However you can drop them by changing the errors parameter.

I also got rid of some of the functions you called as they don't need to be there.

df = pd.DataFrame({'id' : [1, 2, 3], 
                   'timestamp' : ['2019-07-01T21:30:20Z', 
                                  '2999-12-31T21:30:20Z', 
                                  '9999-12-30T21:30:20Z']})

df['timestamp'] = pd.to_datetime(df['timestamp'], 
                                 format="%Y-%m-%dT%H:%M:%SZ", errors='coerce', utc=True)

df['timestamp'] = df['timestamp'].dt.tz_convert('Europe/Prague')

df.dropna() if you need to drop the NaT values