[英]How to efficiently resample a DatetimeIndex
Pandas has a resample
method on a series/dataframe but there seems no way to resample a DatetimeIndex
on its own? Pandas在一个系列/数据帧上有一个
resample
方法,但似乎没有办法自己重新采样DatetimeIndex
?
Concretely, I have a daily Datetimeindex
with possibly missing dates and I want to resample it at an hourly freq but only including days which are in the original daily index. 具体来说,我有一个每日
Datetimeindex
可能缺少日期,我想以小时频率重新取样,但只包括原始每日索引中的天数。
Is there a better way than my attempt below? 有没有比我下面尝试更好的方法?
In [56]: daily_index = pd.period_range('01-Jan-2017', '31-Jan-2017', freq='B').asfreq('D')
In [57]: daily_index
Out[57]:
PeriodIndex(['2017-01-02', '2017-01-03', '2017-01-04', '2017-01-05',
'2017-01-06', '2017-01-09', '2017-01-10', '2017-01-11',
'2017-01-12', '2017-01-13', '2017-01-16', '2017-01-17',
'2017-01-18', '2017-01-19', '2017-01-20', '2017-01-23',
'2017-01-24', '2017-01-25', '2017-01-26', '2017-01-27',
'2017-01-30', '2017-01-31'],
dtype='int64', freq='D')
In [58]: daily_index.shape
Out[58]: (22,)
In [59]: hourly_index = pd.DatetimeIndex([]).union_many(
...: pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
...: for day in daily_index
...: )
In [60]: hourly_index
Out[60]:
DatetimeIndex(['2017-01-02 00:00:00', '2017-01-02 01:00:00',
'2017-01-02 02:00:00', '2017-01-02 03:00:00',
'2017-01-02 04:00:00', '2017-01-02 05:00:00',
'2017-01-02 06:00:00', '2017-01-02 07:00:00',
'2017-01-02 08:00:00', '2017-01-02 09:00:00',
...
'2017-01-31 14:00:00', '2017-01-31 15:00:00',
'2017-01-31 16:00:00', '2017-01-31 17:00:00',
'2017-01-31 18:00:00', '2017-01-31 19:00:00',
'2017-01-31 20:00:00', '2017-01-31 21:00:00',
'2017-01-31 22:00:00', '2017-01-31 23:00:00'],
dtype='datetime64[ns]', length=528, freq=None)
In [61]: 22*24
Out[61]: 528
In [62]: %%timeit
...: hourly_index = pd.DatetimeIndex([]).union_many(
...: pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
...: for day in daily_index
...: )
100 loops, best of 3: 13.7 ms per loop
UPDATE: 更新:
I went with a slight variation of @NTAWolf's answer which has similar performance but won't reorder the input dates in case they're not sorted 我和@ NTAWolf的答案略有不同,它有相似的性能,但是如果它们没有排序,它们不会重新排序输入日期
def resample_index(index, freq):
"""Resamples each day in the daily `index` to the specified `freq`.
Parameters
----------
index : pd.DatetimeIndex
The daily-frequency index to resample
freq : str
A pandas frequency string which should be higher than daily
Returns
-------
pd.DatetimeIndex
The resampled index
"""
assert isinstance(index, pd.DatetimeIndex)
start_date = index.min()
end_date = index.max() + pd.DateOffset(days=1)
resampled_index = pd.date_range(start_date, end_date, freq=freq)[:-1]
series = pd.Series(resampled_index, resampled_index.floor('D'))
return pd.DatetimeIndex(series.loc[index].values)
In [184]: %%timeit
...: hourly_index3 = pd.date_range(daily_index.start_time.min(),
...: daily_index.end_time.max() + 1,
...: normalize=True, freq='H')
...: hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]
100 loops, best of 3: 2.97 ms per loop
In [185]: %timeit resample_index(daily_index.to_timestamp('D','S'), freq='H')
100 loops, best of 3: 2.93 ms per loop
| Method | Time | Relative | |---------------------------------|---------|----------| | OP's updated approach | 1.31 ms | 17.6 % | | Generate daterange, np.in1d | 1.75 ms | 23.5 % | | Generate daterange, Series.isin | 1.90 ms | 25.5 % | | Resample with dummy Series | 4.37 ms | 58.7 % | | OP's initial approach | 7.45 ms | 100.0 % |
np.in1d
np.in1d
Again, @IanS inspired more optimisation! 再次,@ IanS激发了更多优化! This is a little less readable, but a bit faster:
这可读性稍差,但速度要快一些:
%%timeit -r 10
hourly_index4 = pd.date_range(daily_index.start_time.min(),
daily_index.end_time.max() + pd.DateOffset(days=1),
normalize=True, freq='H')
overlap = np.in1d(np.array(hourly_index4.values, dtype='datetime64[D]'),
np.array(daily_index.start_time.values, dtype='datetime64[D]'))
hourly_index4 = hourly_index4[overlap]
1000 loops, best of 10: 1.75 ms per loop
Here, speedups are gained by converting the values of both Series to the same numpy datetime kind (flooring hourly_index
in the process). 在这里,通过将两个系列的值转换为相同的numpy日期时间类型(过程中的地板
hourly_index
)来获得加速。 Passing .values
to numpy sped it up a little bit. 传递
.values
到numpy加速了一点点。
Series.isin
Series.isin
Faster approach than the initial bid, inspired by @IanS's approach: Generate daterange for the complete range of dates in your data, per hour, and select only those entries that match existing dates in your data: 受@ IanS方法启发,比初始出价更快的方法:为您的数据中的每小时生成完整日期范围的日期范围,并仅选择与数据中现有日期匹配的条目:
%%timeit
hourly_index3 = pd.date_range(daily_index.start_time.min(),
# The following line should use
# +pd.DateOffset(days=1) in place of +1
# but is left as is to show the option.
daily_index.end_time.max() + 1,
normalize=True, freq='H')
hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]
100 loops, best of 3: 1.9 ms per loop
which cuts off about 75% processing time. 这减少了大约75%的处理时间。
Using a dummy series, you can avoid the looping. 使用虚拟系列,可以避免循环。 On my computer, it cuts off about 40% of the run time.
在我的电脑上,它减少了大约40%的运行时间。
I get the following time for your approach: 我得到以下时间用于您的方法:
In [14]: %%timeit -o -r 10
....: hourly_index = pd.DatetimeIndex([]).union_many(
....: pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
....: for day in daily_index
....: )
....:
100 loops, best of 10: 7.45 ms per loop
And for the faster approach: 而对于更快的方法:
In [13]: %%timeit -o -r 10
s = pd.Series(0, index=daily_index)
s = s.resample('H').last()
s = s[s.index.start_time.floor('D').isin(daily_index.start_time)]
hourly_index2 = s.index.start_time
....:
100 loops, best of 10: 4.37 ms per loop
Note that we don't really care about the value put into the series; 请注意,我们并不关心系列中的价值; here I just default to
int
. 这里我只是默认为
int
。
The expression s.index.start_time.floor('D').isin(daily_index.start_time)
gives us a boolean vector for which values in s.index
that match days in daily_index
. 表达
s.index.start_time.floor('D').isin(daily_index.start_time)
为我们提供了其中在值的布尔矢量s.index
在匹配天即daily_index
。
Another option would be to generate the hourly index directly, and remove non-business days afterwards: 另一种选择是直接生成每小时索引,然后删除非工作日:
hourly_index = pd.date_range('01-Jan-2017', '31-Jan-2017', freq='H')
hourly_index = hourly_index[hourly_index.dayofweek < 5]
Performance comparison: 表现比较:
10 loops, best of 3: 44.2 ms per loop
10 loops, best of 3: 44.2 ms per loop
1000 loops, best of 3: 1.46 ms per loop
1000 loops, best of 3: 1.46 ms per loop
1000 loops, best of 3: 598 µs per loop
1000 loops, best of 3: 598 µs per loop
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.