[英]How to resample daily data to hourly data for all whole days with pandas?
我有一个 dataframe df,如下所示:
city datetime value
0 city_a 2020-07-10 2
1 city_a 2020-07-11 5
2 city_b 2020-07-11 4
我正在尝试以 6 小时的频率重新采样每日日期时间(每 00 小时、6 小时、12 小时和 18 小时的数据)。
下面的代码给了我几乎我期待的 output
my_df = my_df.set_index(['datetime', 'city'])
my_df = my_df.unstack(-1).resample('6H').pad()
my_df = my_df.stack().reset_index()
my_df = my_df[['city', 'datetime', 'value']]
my_df = my_df.sort_values(['city', 'datetime'])
Output:
city datetime value
0 city_a 2020-07-10 00:00:00 2.0
1 city_a 2020-07-10 06:00:00 2.0
2 city_a 2020-07-10 12:00:00 2.0
3 city_a 2020-07-10 18:00:00 2.0
4 city_a 2020-07-11 00:00:00 5.0
5 city_b 2020-07-11 00:00:00 4.0
但是,我们可以看到 2020-07-11 的日子并不完整。 我希望包括 2020-07-11 06:00:00、12:00:00 和 18:00:00 在内的行出现在 output 中。
所以我预期的 output 应该是:
city datetime value
0 city_a 2020-07-10 00:00:00 2.0
1 city_a 2020-07-10 06:00:00 2.0
2 city_a 2020-07-10 12:00:00 2.0
3 city_a 2020-07-10 18:00:00 2.0
4 city_a 2020-07-11 00:00:00 5.0
6 city_a 2020-07-11 06:00:00 5.0
8 city_a 2020-07-11 12:00:00 5.0
10 city_a 2020-07-11 18:00:00 5.0
5 city_b 2020-07-11 00:00:00 4.0
7 city_b 2020-07-11 06:00:00 4.0
9 city_b 2020-07-11 12:00:00 4.0
11 city_b 2020-07-11 18:00:00 4.0
Pandas 有没有一种优雅的方法?
生成 dataframe 的代码:
my_df = pd.DataFrame(data = {
'city': ['city_a', 'city_a', 'city_b'],
'datetime':
[pd.to_datetime('2020/07/10'),pd.to_datetime('2020/07/11'),pd.to_datetime('2020/07/11')],
'value': [2,5,4]
})
使用:
# STEP A
df1 = (df.groupby('city')['datetime'].max() + pd.Timedelta(days=1)).reset_index()
# STEP B
df1 = pd.concat([df, df1]).set_index('datetime')
# STEP C
df1 = df1.groupby('city', as_index=False).resample('6H').ffill()
# STEP D
df1 = df1.reset_index().drop('level_0', 1).dropna(subset=['value'])
细节:
步骤 A:使用DataFrame.groupby
对city
的 dataframe 进行分组,以确定每组中日期的最大值,并将每组的最大值加1 day
,这将需要重新采样 Z6A8064B3DF47945557D5C。
# print(df1)
city datetime
0 city_a 2020-07-12
1 city_b 2020-07-12
STEP B: Using pd.concat
to concat the original dataframe df
to the newly created dataframe df1
, this is because we have to resample the dataframe in STEP C.
# print(df1)
city value
datetime
2020-07-10 city_a 2.0
2020-07-11 city_a 5.0
2020-07-11 city_b 4.0
2020-07-12 city_a NaN
2020-07-12 city_b NaN
STEP C:使用DataFrame.resample
重新采样 dataframe 以6H
的频率分组在city
上,并使用ffill
向前填充值。
# print(df1)
city value
datetime
0 2020-07-10 00:00:00 city_a 2.0
2020-07-10 06:00:00 city_a 2.0
2020-07-10 12:00:00 city_a 2.0
2020-07-10 18:00:00 city_a 2.0
2020-07-11 00:00:00 city_a 5.0
2020-07-11 06:00:00 city_a 5.0
2020-07-11 12:00:00 city_a 5.0
2020-07-11 18:00:00 city_a 5.0
2020-07-12 00:00:00 city_a NaN
1 2020-07-11 00:00:00 city_b 4.0
2020-07-11 06:00:00 city_b 4.0
2020-07-11 12:00:00 city_b 4.0
2020-07-11 18:00:00 city_b 4.0
2020-07-12 00:00:00 city_b NaN
步骤 D:最后使用DataFrame.reset_index
并使用DataFrame.drop
沿axis=1
删除未使用的列,还使用DataFrame.dropna
删除具有NaN
value
的行中的列
# print(df1)
datetime city value
0 2020-07-10 00:00:00 city_a 2.0
1 2020-07-10 06:00:00 city_a 2.0
2 2020-07-10 12:00:00 city_a 2.0
3 2020-07-10 18:00:00 city_a 2.0
4 2020-07-11 00:00:00 city_a 5.0
5 2020-07-11 06:00:00 city_a 5.0
6 2020-07-11 12:00:00 city_a 5.0
7 2020-07-11 18:00:00 city_a 5.0
9 2020-07-11 00:00:00 city_b 4.0
10 2020-07-11 06:00:00 city_b 4.0
11 2020-07-11 12:00:00 city_b 4.0
12 2020-07-11 18:00:00 city_b 4.0
我看到的唯一方法是添加一个空行,其日期时间等于最新的现有日期时间 + 一天。 然后你几乎可以做同样的事情(pivot 是替换 set_index 和 unstack 的便捷方法)。
# adding a row where datetime corresponds to the max datetime + 1 day
df.loc[len(df), 'datetime'] = df.datetime.max() + pd.Timedelta(days=1)
# pivot to replace set_index & unstack
df = (df.pivot(index='datetime', columns='city')
.resample('6H')
.pad(3)
.stack()
.reset_index()
.sort_values(['city', 'datetime']))
df[['city', 'datetime', 'value']]
city datetime value
0 city_a 2020-07-10 00:00:00 2.0
1 city_a 2020-07-10 06:00:00 2.0
2 city_a 2020-07-10 12:00:00 2.0
3 city_a 2020-07-10 18:00:00 2.0
4 city_a 2020-07-11 00:00:00 5.0
6 city_a 2020-07-11 06:00:00 5.0
8 city_a 2020-07-11 12:00:00 5.0
10 city_a 2020-07-11 18:00:00 5.0
5 city_b 2020-07-11 00:00:00 4.0
7 city_b 2020-07-11 06:00:00 4.0
9 city_b 2020-07-11 12:00:00 4.0
11 city_b 2020-07-11 18:00:00 4.0
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.