Python Pandas to_datetime Out of bounds nanosecond timestamp on a pandas.datetime

Question

I am using Python 2--I am behind moving over my code--so perhaps this issue has gone away. Using pandas, I can create a datetime like this:

import pandas as pd
big_date= pd.datetime(9999,12,31)
print big_date
9999-12-31 00:00:00
big_date2 = pd.to_datetime(big_date)
 . . .
Out of bounds nanosecond timestamp: 9999-12-31 00:00:00

I understand the reason for the error in that there are obviously too many nanoseconds in a date that big. I also know that big_date2 = pd.to_datetime(big_date, errors='ignore') would work. However, in my situation, I have a column of what are supposed to be dates (read from SQL server) and I do indeed want it to change invalid data/dates to NaT. In effect, I was using pd.to_datetime as a validity check. To Pandas, on the one hand, 9999-12-31 is a valid date, and on the other, it's not. That means I can't use it and have had to come up with something else.

I've played around with the arguments in pandas to_datetime and not been able to solve this.

I've looked at other questions/problems of this nature, and not found an answer.

Answer 1

I have a similar issue and was able to find a solution.

I have a pandas dataframe with one column that contains a datetime (retrieved from a database table where the column was a DateTime2 data type), but I need to be able to represents date that are further in the future than the Timestamp.max value.

Fortunately, I didn't need to worry about the time part of the datetime column - it was actually always 00:00:00 (I didn't create the database design and, yes, it probably should have been a Date data type and not a DateTime2 data type). So I was able to get round the issue by converting the pandas dataframe column to just a date type. For example:

for i, row in df.iterrows():
    df.set_value(i, 'DateColumn', datetime.datetime(9999, 12, 31).date())

sets all of the values in the column to the date 9999-12-31 and you don't receive any errors when using this column anymore.

So, if you can afford to lose the time part of the date you are trying to use you can work round the limitation of the datetime values in the dataframe by converting to a date.