通过转换函数增量添加到 Pandas Group

Question

I have a large DataFrame with many columns that are GroupBy functions of the original data. Computing all these functions takes a long time. Each day I get some new data and currently I compute all these functions from scratch. Is there a way to do these GroupBy functions without having to compute the whole functions again. I will provide a small DataFrame as an example:

df = pd.DataFrame({'x': [0, 1, 2, 5, 4, 5, 8, 7], 'g1': ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'a'], 'g2': ['a', 'b', 'a', 'a', 'b', 'b', 'a', 'a']})

   x g1 g2
0  0  a  a
1  1  b  b
2  2  c  a
3  5  a  a
4  4  b  b
5  5  c  b
6  8  a  a
7  7  a  a

Now an example column:

def lag(array):
    out = np.nan * array
    out[1:] = array[:-1]
    return out

df['y'] = df.groupby(['g1', 'g2'])['x'].transform(lag)

   x g1 g2    y
0  0  a  a  NaN
1  1  b  b  NaN
2  2  c  a  NaN
3  5  a  a  0.0
4  4  b  b  1.0
5  5  c  b  NaN
6  8  a  a  5.0
7  7  a  a  8.0

Now let's say I get some new data to append to my original DataFrame:

newdf = pd.DataFrame({'x': [2, 1], 'g1': ['a', 'b'], 'g2': ['a', 'b']})
df = df.append(newdf)

   x g1 g2    y
0  0  a  a  NaN
1  1  b  b  NaN
2  2  c  a  NaN
3  5  a  a  0.0
4  4  b  b  1.0
5  5  c  b  NaN
6  8  a  a  5.0
7  7  a  a  8.0
0  2  a  a  NaN
1  1  b  b  NaN

Is there now a way to work out 'y' for the last 2 rows without just recalculating the whole column to produce the following DataFrame?

   x g1 g2    y
0  0  a  a  NaN
1  1  b  b  NaN
2  2  c  a  NaN
3  5  a  a  0.0
4  4  b  b  1.0
5  5  c  b  NaN
6  8  a  a  5.0
7  7  a  a  8.0
0  2  a  a  7.0
1  1  b  b  4.0

Answer 1

One way of doing this is to do this:

Create first a column that indicates which rows lag has been applied to, then apply lag to the rows that haven't by using mask

df['applied'] = 1
df = df.append(newdf)
df['y'].mask(df['applied']!=1, df.groupby(['g1', 'g2'])['x'].transform(lag), inplace=True)

which gives

 x g1 g2    y  applied
0  0  a  a  NaN      1.0
1  1  b  b  NaN      1.0
2  2  c  a  NaN      1.0
3  5  a  a  0.0      1.0
4  4  b  b  1.0      1.0
5  5  c  b  NaN      1.0
6  8  a  a  5.0      1.0
7  7  a  a  8.0      1.0
0  2  a  a  7.0      NaN
1  1  b  b  4.0      NaN

then drop the applied column:

df = df.drop(['applied'], axis=1)

which gives you what you wanted:

x g1 g2    y
0  0  a  a  NaN
1  1  b  b  NaN
2  2  c  a  NaN
3  5  a  a  0.0
4  4  b  b  1.0
5  5  c  b  NaN
6  8  a  a  5.0
7  7  a  a  8.0
0  2  a  a  7.0
1  1  b  b  4.0
​```