如何让 Pandas 将包含 NaT 的列从 timedelta 转换为 datetime？

Question

I have a pandas dataframe with a column which is of type timedelta64[ns] , and which I would like to convert ot datetime64[ns] .

The pd.to_datetime() function purports to do just that, and has worked in the past, but appears to fail now. I would assume this might be related to an API quirk which has gone beneath my radar. Currently it fails with:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/site-packages/pandas/core/tools/datetimes.py", line 724, in to_datetime
    cache_array = _maybe_cache(arg, format, cache, convert_listlike)
  File "/usr/lib/python3.7/site-packages/pandas/core/tools/datetimes.py", line 152, in _maybe_cache
    cache_dates = convert_listlike(unique_dates, format)
  File "/usr/lib/python3.7/site-packages/pandas/core/tools/datetimes.py", line 363, in _convert_listlike_datetimes
    arg, _ = maybe_convert_dtype(arg, copy=False)
  File "/usr/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 1916, in maybe_convert_dtype
    raise TypeError(f"dtype {data.dtype} cannot be converted to datetime64[ns]")
TypeError: dtype timedelta64[ns] cannot be converted to datetime64[ns]

To try and reproduce, please use the MWE below:

wget https://chymera.eu/ppb/61ebad.csv
python
import pandas as pd
df = pd.read_csv('61ebad.csv')
df['Animal_death_date'] = pd.to_timedelta(df['Animal_death_date'], errors='coerce')
df['Animal_death_date'] = pd.to_datetime(df['Animal_death_date'], errors='coerce')

The error also occurs if I am using errors='ignore' . For reference, I am using Pandas 1.0.1 .

Answer 1

If need convert timedeltas to datetime, add some start datetime:

import pandas as pd

df = pd.read_csv('https://chymera.eu/ppb/61ebad.csv')
start = pd.to_datetime('2000-01-01')
df['Animal_death_date'] = pd.to_timedelta(df['Animal_death_date'], errors='coerce') + start
print (df['Animal_death_date'] )
0                     NaT
1                     NaT
2                     NaT
3                     NaT
4                     NaT

843                   NaT
844                   NaT
845   2000-05-12 19:00:00
846   2000-05-12 19:00:00
847   2000-05-12 19:00:00
Name: Animal_death_date, Length: 848, dtype: datetime64[ns]

Or add some column filled by datetimes:

import pandas as pd

df = pd.read_csv('https://chymera.eu/ppb/61ebad.csv')
start = pd.to_datetime(df['FMRIMeasurement_date'])
df['Animal_death_date'] = pd.to_timedelta(df['Animal_death_date'], errors='coerce') + start
print (df['Animal_death_date'] )
0                     NaT
1                     NaT
2                     NaT
3                     NaT
4                     NaT

843                   NaT
844                   NaT
845   2018-10-04 19:20:54
846   2018-10-04 19:20:54
847   2018-10-04 19:20:54
Name: Animal_death_date, Length: 848, dtype: datetime64[ns]

Answer 2

Start with a small correction: Your source column in question is also a text column, but only formatted as timedelta .

To convert Animal_death_date column define the following function:

def myDateConv(tt):
    return pd.to_datetime('2020-' + tt, format='%Y-%j days %X.%f')\
        if len(tt) > 0 else np.nan

I assume that your dates are from this year, hence 2020 as the initial part of the whole date string. If they are from other year, change this prefix accordingly.

But apply this function as early as when you read the source file:

df = pd.read_csv('61ebad.csv', index_col=0, parse_dates=['Treatment_start_date',
    'Treatment_end_date', 'FMRIMeasurement_date', 'OpenFieldTestMeasurement_date',
    'ForcedSwimTestMeasurement_date', 'CageStay_start_date', 'Cage_Treatment_start_date',
    'Cage_Treatment_end_date', 'SucrosePreferenceMeasurement_date', 'reference_date'],
    converters = { 'Animal_death_date': myDateConv })

Note additional parameters:

• index_col - to treat the initial column as the index,
• parse_dates - to convert "normally" formatted dates to datetime ,
• converters - to apply the above function to the source of Animal_death_date column.

I think, this solution is simpler and more readable than individual conversion of particular columns.