将带有包含 timedelta64 对象列表的列的 DataFrame 保存为 Parquet 文件

Question

I have a list of lists of Pandas timedelta64 called td_list like

[Timedelta('0 days 01:06:15'), Timedelta('0 days 01:34:46'), Timedelta('0 days 00:00:00'), Timedelta('0 days 00:00:00')]
[Timedelta('0 days 01:51:46'), Timedelta('0 days 01:40:40')]
[Timedelta('0 days 07:07:52'), Timedelta('0 days 07:32:00'), Timedelta('0 days 00:00:00'), Timedelta('0 days 04:54:26')]
[Timedelta('0 days 00:00:00'), Timedelta('0 days 04:28:36'), Timedelta('0 days 10:49:42'), Timedelta('0 days 06:36:23')]

I'm appending it as a DataFrame new column

df["td_col"] = td_list 

and everything is fine, I obtain a column with elements

[0 days 01:06:15, 0 days 01:34:46, 0 days 00:00:00, 0 days 00:00:00]
[0 days 01:51:46, 0 days 01:40:40]

etc...

as expected. but when I'm saving it with

df.to_parquet(path,compression=None, engine="fastparquet")

I obtain

Can't infer object conversion type: 0    0   0 days 02:58:20.333333333
dtype: timedelta...
1    0   0 days 02:05:58.727272727
dtype: timedelta...
2    0   0 days 01:45:38.250000
dtype: timedelta64[ns]
3    0   0 days 00:40:15.250000
dtype: timedelta64[ns]
4           0   0 days 01:46:13
dtype: timedelta64[ns]
5    0   0 days 04:53:34.500000
dtype: timedelta64[ns]
6    0   0 days 05:28:40.250000
dtype: timedelta64[ns]
7    0   0 days 02:23:05.500000
dtype: timedelta64[ns]
8    0   0 days 01:50:01.500000
dtype: timedelta64[ns]
9                    0   0 days
dtype: timedelta64[ns]
Name: td_col, dtype: object

Do you know how can I fix this?

Answer 1

Even though your values are pandas datetime values, your column d-type is object. You should convert your date column to a datetime column first using:

df["td_col"] = pd.to_datetime(df["td_col"])

Answer 2

try using a different engine. I tried engine="pyarrow" for a column with list and it worked