大熊猫根据列分组的指标计算差异

Question

Here is my question. I don't know how to describe it, so I will just give an example.

a b k
0 0 0
0 1 1
0 2 0
0 3 0
0 4 1
0 5 0
1 0 0
1 1 1
1 2 0
1 3 1
1 4 0

Here, "a" is user id, "b" is time, and "k" is a binary indicator flag. "b" is consecutive for sure. What I want to get is this:

a b k diff_b
0 0 0 nan
0 1 1 nan
0 2 0 1
0 3 0 2
0 4 1 3
0 5 0 1
1 0 0 nan
1 1 1 nan
1 2 0 1
1 3 1 2
1 4 0 1

So, diff_b is a time difference variable. It shows the duration between the current time point and the last time point with an action. If there is never an action before, it returns nan. This diff_b is grouped by a. For each user, this diff_b is calculated independently.

Can anyone revise my title? I don't know how to describe it in english. So complex...

Thank you!

Answer 1

IIUC

df['New']=df.b.loc[df.k==1]# get all value b when k equal to 1
df.New=df.groupby('a').New.apply(lambda x : x.ffill().shift()) # fillna by froward method , then we need shift.
df.b-df['New']# yield 
Out[260]: 
0     NaN
1     NaN
2     1.0
3     2.0
4     3.0
5     1.0
6     NaN
7     NaN
8     1.0
9     2.0
10    1.0
dtype: float64

Answer 2

create partitions of the data of rows after k == 1 up to the next k == 1 using cumsum, and shift, for each group of a

parts = df.groupby('a').k.apply(lambda x: x.shift().cumsum())

group by the df.a & parts and calculate the difference between b & b.min() within each group

vals = df.groupby([df.a, parts]).b.apply(lambda x: x-x.min()+1)

set values to null when part == 0 & assign back to the dataframe

df['diff_b'] = np.select([parts!=0], [vals], np.nan)

outputs:

    a  b  k  diff_b
0   0  0  0     NaN
1   0  1  1     NaN
2   0  2  0     1.0
3   0  3  0     2.0
4   0  4  1     3.0
5   0  5  0     1.0
6   1  0  0     NaN
7   1  1  1     NaN
8   1  2  0     1.0
9   1  3  1     2.0
10  1  4  0     1.0