在熊猫指数级别的Groupby

Question

I have the following MultiIndex DataFrame and I'm wondering if there is a way to apply different functions on the second level index.

import pandas as pd
# Creation
df1 = pd.DataFrame([[1,2,1],[4,5,1],[4,5,2]], columns=["M1","M2","month"])
df1['var']="v1"
df2 = pd.DataFrame([[1.5,2.5,1],[4.5,5.5,1],[1.5,1.5,2]], columns=["M1","M2","month"])
df2['var']="v2"
df_all = pd.concat([df1,df2],join='outer')

# Final DataFrame
df_all_idx = df_all.set_index(["month","var"],inplace=False)
df_all_idx.sort_index(level=[0])

                M1  M2
month    var        
1        v1     1.0 2.0
         v1     4.0 5.0
         v2     1.5 2.5
         v2     4.5 5.5
2        v1     4.0 5.0
         v2     1.5 1.5

With groupby I can obtain:

df_grp = df_all_idx.groupby(by=["month","var"]).sum()

                M1  M2
month   var     
1       v1      5.0 7.0
        v2      6.0 8.0
2       v1      4.0 5.0
        v2      1.5 1.5

For example, I would need to apply sum() to v1 values and a custom function to v2 values.

Thanks

Answer 1

I like dictionaries. So I would store your aggregating functions in a dictionary, and look them up based on each group's name.

import numpy
import pandas

aggregators = {
    'v2': numpy.min
}


df1 = pandas.DataFrame(
    [[1, 2, 1],[4, 5, 1],[4, 5, 2]],
    columns=["M1", "M2", "month"]
).assign(var='v1')

df2 = pandas.DataFrame(
    [[1.5,2.5,1], [4.5,5.5,1], [1.5,1.5,2]],
    columns=["M1", "M2", "month"]
).assign(var='v2')

df = (
    pandas.concat([df1, df2], join='outer')
        .groupby(by=['month', 'var'])
        .apply(lambda g: aggregators.get(g.name[-1], numpy.sum)(g))
        [['M1', 'M2']]
)

And that's:

            M1   M2
month var          
1     v1   5.0  7.0
      v2   1.5  2.5
2     v1   4.0  5.0
      v2   1.5  1.5

This line: .apply(lambda g: aggregators.get(g.name[-1], numpy.sum)(g)) is a little complicated. Here's what it does:

1. .apply loops through all of the groups and runs them through the lambda
2. Each group has a name attribute that is the values of grouping columns
3. g.name[-1] is the last element (v1, v2)
4. aggregators.get(g.name[-1], numpy.sum) looks up the function to use, but if a function can't be found, it defaults to numpy.sum
5. then we pass the group to the function that we looked up

Answer 2

Would something like this work?

df_all_idx.xs('v1', level=1).sum(axis=1)
df_all_idx.xs('v2', level=1).apply(some_function, axis=1)

Answer 3

Following the suggestion to split, apply and concat back I came up with this solution:

def myfunc(x):
    return np.mean(x)

p1 = df_all_idx.loc[(slice(None), 'v1'), :].groupby(by=["month","var"]).sum()    
p2 = df_all_idx.loc[(slice(None), 'v2'), :].groupby(by=["month","var"]).agg(myfunc)

pd.concat([p1,p2], join='outer').sort_index(level=[0])

That return the result as I want it:

                 M1     M2
 month  var     
 1       v1      5.0    7.0
         v2      3.0    4.0
 2       v1      4.0    5.0
         v2      1.5    1.5

I assume then that this is the best practice in this case.