[英]Pandas filling missing dates and values within group
I've a data frame that looks like the following我有一个如下所示的数据框
x = pd.DataFrame({'user': ['a','a','b','b'], 'dt': ['2016-01-01','2016-01-02', '2016-01-05','2016-01-06'], 'val': [1,33,2,1]})
What I would like to be able to do is find the minimum and maximum date within the date column and expand that column to have all the dates there while simultaneously filling in 0
for the val
column.我想做的是在日期列中找到最小和最大日期,并扩展该列以包含所有日期,同时为
val
列填充0
。 So the desired output is所以所需的 output 是
dt user val
0 2016-01-01 a 1
1 2016-01-02 a 33
2 2016-01-03 a 0
3 2016-01-04 a 0
4 2016-01-05 a 0
5 2016-01-06 a 0
6 2016-01-01 b 0
7 2016-01-02 b 0
8 2016-01-03 b 0
9 2016-01-04 b 0
10 2016-01-05 b 2
11 2016-01-06 b 1
I've tried the solution mentioned here and here but they aren't what I'm after.我已经尝试过这里和这里提到的解决方案,但它们不是我所追求的。 Any pointers much appreciated.
任何指针都非常感谢。
Initial Dataframe:初始数据框:
dt user val
0 2016-01-01 a 1
1 2016-01-02 a 33
2 2016-01-05 b 2
3 2016-01-06 b 1
First, convert the dates to datetime:首先,将日期转换为日期时间:
x['dt'] = pd.to_datetime(x['dt'])
Then, generate the dates and unique users:然后,生成日期和唯一用户:
dates = x.set_index('dt').resample('D').asfreq().index
>> DatetimeIndex(['2016-01-01', '2016-01-02', '2016-01-03', '2016-01-04',
'2016-01-05', '2016-01-06'],
dtype='datetime64[ns]', name='dt', freq='D')
users = x['user'].unique()
>> array(['a', 'b'], dtype=object)
This will allow you to create a MultiIndex:这将允许您创建一个 MultiIndex:
idx = pd.MultiIndex.from_product((dates, users), names=['dt', 'user'])
>> MultiIndex(levels=[[2016-01-01 00:00:00, 2016-01-02 00:00:00, 2016-01-03 00:00:00, 2016-01-04 00:00:00, 2016-01-05 00:00:00, 2016-01-06 00:00:00], ['a', 'b']],
labels=[[0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]],
names=['dt', 'user'])
You can use that to reindex your DataFrame:您可以使用它来重新索引您的 DataFrame:
x.set_index(['dt', 'user']).reindex(idx, fill_value=0).reset_index()
Out:
dt user val
0 2016-01-01 a 1
1 2016-01-01 b 0
2 2016-01-02 a 33
3 2016-01-02 b 0
4 2016-01-03 a 0
5 2016-01-03 b 0
6 2016-01-04 a 0
7 2016-01-04 b 0
8 2016-01-05 a 0
9 2016-01-05 b 2
10 2016-01-06 a 0
11 2016-01-06 b 1
which then can be sorted by users:然后可以按用户排序:
x.set_index(['dt', 'user']).reindex(idx, fill_value=0).reset_index().sort_values(by='user')
Out:
dt user val
0 2016-01-01 a 1
2 2016-01-02 a 33
4 2016-01-03 a 0
6 2016-01-04 a 0
8 2016-01-05 a 0
10 2016-01-06 a 0
1 2016-01-01 b 0
3 2016-01-02 b 0
5 2016-01-03 b 0
7 2016-01-04 b 0
9 2016-01-05 b 2
11 2016-01-06 b 1
As @ayhan suggests正如@ayhan 建议的那样
x.dt = pd.to_datetime(x.dt)
One-liner using mostly @ayhan's ideas while incorporating stack
/ unstack
and fill_value
单线主要使用@ayhan 的想法,同时结合
stack
/ unstack
和fill_value
x.set_index(
['dt', 'user']
).unstack(
fill_value=0
).asfreq(
'D', fill_value=0
).stack().sort_index(level=1).reset_index()
dt user val
0 2016-01-01 a 1
1 2016-01-02 a 33
2 2016-01-03 a 0
3 2016-01-04 a 0
4 2016-01-05 a 0
5 2016-01-06 a 0
6 2016-01-01 b 0
7 2016-01-02 b 0
8 2016-01-03 b 0
9 2016-01-04 b 0
10 2016-01-05 b 2
11 2016-01-06 b 1
An old question, with already excellent answers;一个老问题,已经有了很好的答案; this is an alternative, using the complete function from pyjanitor that could help with the abstraction when generating explicitly missing rows:
这是一种替代方法,使用来自pyjanitor的完整函数,它可以在生成显式缺失的行时帮助抽象:
#pip install pyjanitor
import pandas as pd
import janitor as jn
x['dt'] = pd.to_datetime(x['dt'])
# generate complete list of dates
dates = dict(dt = pd.date_range(x.dt.min(), x.dt.max(), freq='1D'))
# build the new dataframe, and fill nulls with 0
x.complete('user', dates, fill_value = 0)
user dt val
0 a 2016-01-01 1
1 a 2016-01-02 33
2 a 2016-01-03 0
3 a 2016-01-04 0
4 a 2016-01-05 0
5 a 2016-01-06 0
6 b 2016-01-01 0
7 b 2016-01-02 0
8 b 2016-01-03 0
9 b 2016-01-04 0
10 b 2016-01-05 2
11 b 2016-01-06 1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.