Pandas.DataFrame.GroupBy.agg，聚合中需要独立的列function。 如何将其放入 agg 中？

Question

I have a Pandas DataFrame object with a two-level MultiIndex. Furthermore it obviously contains a number of additional columns (eg 'A', 'B', 'C', 'D', 'E'). I want to execute some aggregation function on the individual multi indices in the DataFrame on each individual column from a subset of the available columns (say, 'C', 'D', 'E'). For this purpose I select only the subset of columns, use GroupBy to group the thus sliced data frame by levels=[0,1] and execute agg with a dictionary configuring the aggregation function for each of the selected columns from the mentioned subset.

df[['C', 'D', 'E']].groupby(level=[0, 1]).agg({'C': aggfunc, 'D': aggfunc, 'E': aggfunc})

My problem now is that in addition to the currently aggregated column that is handed into the aggregation function, I need a second column, eg 'B', in the aggregation function. So it's basically an aggregation of two columns, one of ['C', 'D', 'E'] plus 'B'.

What I could do is replacing aggfunc with a closure that knows 'B'. Is that the only way? Or is there a way to tell Pandas to also hand 'B' into the aggregation function in addition to 'C', 'D', 'E'?

Example notebook

I've created a Jupyter Notebook to generate example data. In the example, you can see the columns serial and turn which form the MultiIndex, and the column milage which is the independent column that I need in the aggregation function in addition to the columns m1 to m4 each. So in the function I need m<n> (whichever is currently processed) plus milage . Since milage is a float value too I cannot use it as index.

The notebook can be found here: https://github.com/HWiese1980/public_notebooks/blob/master/example.ipynb

Answer 1

Problem is agg function 'see' only processing columns, not another ones.

So it is possible, but not performant, because is necessary filtering per groups:

np.random.seed(2020)
cols = ["serial", "turn", "milage", "m1", "m2", "m3", "m4"]
df = pd.DataFrame(columns=cols).set_index("serial", "turn")

serials = ["11111", "11222", "12345"]

data = []
end = 0.0
for s in serials:
    for t in range(np.random.randint(6)):
        start = end + np.random.rand() * 1000.
        end = start + np.random.rand() * 1000.
        run_point_count = np.random.randint(high=10, low=5)
        milages = np.linspace(start, end, run_point_count)
        for entry in range(run_point_count):
            d = np.hstack((np.array([s, t]), [milages[entry]], np.random.rand(4)))
            _df = {}
            for i, c in enumerate(cols): 
                _df[c] = d[i]
            data.append(_df)

df_out = df.append(data, ignore_index=True, sort=True).set_index(["serial", "turn"])
df_out = df_out.astype(float)
#print (df_out)

---

def aggfunc(x):
    return x.sum() + df_out.loc[x.index, "milage"].mean()

#need unique MultiIndex
df_out = df_out.set_index(df_out.groupby(level=[0, 1]).cumcount(), append=True)
df = (df_out.groupby(level=[0, 1])
           .agg({'m1': aggfunc, 'm2': aggfunc, 'm3': aggfunc, 'm4': aggfunc}))
print (df)
                      m1           m2           m3           m4
serial turn                                                    
12345  0      735.612167   734.425345   733.988098   736.534878
       1     1763.739719  1762.587273  1763.196721  1763.929828
       2     2582.773092  2583.585509  2582.582403  2582.121202

Second solution is with convert column to FloatIndex :

def aggfunc(x):
    return x.sum() + np.mean(x.index.get_level_values(3))


df = (df_out.set_index('milage', append=True)
           .groupby(level=[0, 1])
           .agg({'m1': aggfunc, 'm2': aggfunc, 'm3': aggfunc, 'm4': aggfunc}))
print (df)
                      m1           m2           m3           m4
serial turn                                                    
12345  0      735.612167   734.425345   733.988098   736.534878
       1     1763.739719  1762.587273  1763.196721  1763.929828
       2     2582.773092  2583.585509  2582.582403  2582.121202

EDIT:

If possible use some function working with all column of DataFrame use GroupBy.apply :

def f(x):
       return x[['m1','m2','m3','m4']].sum() + x['milage'].mean()

df = df_out.groupby(level=[0, 1]).apply(f)
print (df)
                      m1           m2           m3           m4
serial turn                                                    
12345  0      735.612167   734.425345   733.988098   736.534878
       1     1763.739719  1762.587273  1763.196721  1763.929828
       2     2582.773092  2583.585509  2582.582403  2582.121202