Pandas Groupby 日期时间列上多列的滚动总和

Question

I am trying to get a rolling sum of multiple columns by group, rolling on a datetime column (ie over a specified time interval). Rolling of one column seems to be working fine, but when I roll over multiple columns by vectorizing, I am getting unexpected results.

My first attempt:

df = pd.DataFrame({"column1": range(6), 
                   "column2": range(6), 
                   'group': 3*['A','B'], 
                   'date':pd.date_range("20190101", periods=6)})

(df.groupby('group').rolling("1d", on='date')['column1'].sum()).groupby('group').shift(fill_value=0)

# output:
group  date      
A      2019-01-01    0.0
       2019-01-03    0.0
       2019-01-05    2.0
B      2019-01-02    0.0
       2019-01-04    1.0
       2019-01-06    3.0
Name: column1, dtype: float64

The above produced the desired results, however I lost the original index in the process. Since in my data some dates are the same, I would have to join back on the original dataframe on group+date which is inefficient. I therefore applied the following to avoid this and to keep the original index:

df.groupby('group').apply(lambda x: x.rolling("1d", on='date')['column1'].sum().shift(fill_value=0))

# output:
group   
A      0    0.0
       2    0.0
       4    2.0
B      1    0.0
       3    1.0
       5    3.0
Name: column1, dtype: float64

With this I can easily assign it to a new column of the original df by sorting on the index. Now I would like to repeat same for 'column2' and do this by vectorization. However, the result I get is unexpected:

df.groupby('group').apply(lambda x: x.rolling("1d", on='date')[['column1','column2']].sum().shift(fill_value=0))

# output:

   column1  column2       date
0      0.0      0.0 1970-01-01
1      0.0      0.0 1970-01-01
2      0.0      0.0 2019-01-01
3      1.0      1.0 2019-01-02
4      2.0      2.0 2019-01-03
5      3.0      3.0 2019-01-04

The result is correct, but unexpected for the following reasons: (1) group_keys in the groupby is ignored (2) It sorted the result automatically and reset the index like in a 'transform' method.

I would like to understand why this happened and also are there alternative ways to achieve the results above.

Answer 1

I took your original approach and did some changes. Can you check if this is what you wanted?

Reset the index of the original data frame and assign the original index a column name.

df = df.reset_index().rename(columns={df.index.name: 'index'})

Now, you have the same original data frame, but it has an additional column called index that is the original index.

Apply the rolling on the groupby data frame grouped by group and index columns on the 2 columns column1 and column2 .

(df.groupby(['group', 'index']).rolling("1d", on='date')[['column1', 'column2']].sum()).groupby('group').shift(fill_value=0)

Result:

                        column1  column2
group index date                        
A     0     2019-01-01      0.0      0.0
      2     2019-01-03      0.0      0.0
      4     2019-01-05      2.0      2.0
B     1     2019-01-02      0.0      0.0
      3     2019-01-04      1.0      1.0
      5     2019-01-06      3.0      3.0

If you want the original index back, reset the multi-index and set the 'index' as index

(df.groupby(['group', 'index']).rolling("1d", on='date')[['column1', 'column2']].sum()).groupby('group').shift(fill_value=0).reset_index().set_index('index')

Result:

      group       date  column1  column2
index                                   
0         A 2019-01-01      0.0      0.0
2         A 2019-01-03      0.0      0.0
4         A 2019-01-05      2.0      2.0
1         B 2019-01-02      0.0      0.0
3         B 2019-01-04      1.0      1.0
5         B 2019-01-06      3.0      3.0

Add a .sort_index() if you want it sorted

      group       date  column1  column2
index                                   
0         A 2019-01-01      0.0      0.0
1         B 2019-01-02      0.0      0.0
2         A 2019-01-03      0.0      0.0
3         B 2019-01-04      1.0      1.0
4         A 2019-01-05      2.0      2.0
5         B 2019-01-06      3.0      3.0

Hope this helps. Let me know if I am missing anything.