在 Dask 中使用 read_sql_table 将时间戳分配为索引

Question

In SQLite, I have a table data with index column time which is a timestamp generated at recording by time.time()

I want to load data from this table to a Dask DataFrame. For that I use:

import dask.dataframe as dd
data = dd.read_sql_table('data', 'sqlite:///'+DB_PATH, index_col='time', parse_dates={"time": {"unit":"s"}})

When i want to apply a rolling mean to the data data.resample('15S').mean() , I get:

TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Float64Index'

So if I check the index by data.index , it gives me this, suggesting it's in the right dtype and that the parse_dates actually worked, right? :

Dask Index Structure:
npartitions=1
1.619876e+09    datetime64[ns]
1.620067e+09               ...
Name: time, dtype: datetime64[ns]
Dask Name: from-delayed, 3 tasks

Finally, I tried to defined the index datetime after loading:

data = dd.read_sql_table('data', 'sqlite:///'+DB_PATH, index_col='time')
data['time__1'] = pd.to_datetime(np.array(data['time__1']), unit='s') # By the way, I don't know why Dask creates a 'time__1' column...
data = data.set_index('time__1', sorted=True)

But then I get this message...

ValueError: Length of values does not match length of index

Answer 1

Here is a solution I came up with. It does not seem to be the most efficient since it does not take advantage of date_parse directly from read_sql_table() and rely on column time__1 that was generated by Dask for a certain reason I don't know...

data = dd.read_sql_table('data', 'sqlite:///'+DB_PATH, index_col='time')
data = data.set_index(data['time__1'].map_partitions(pd.to_datetime, unit='s'))
data = data.drop('time__1', axis=1)

If you know a better solution...