具有类别变量组合的GroupBy

Question

Let's say I have data:

pd.DataFrame({'index': ['a','b','c','a','b','c'], 'column': [1,2,3,4,1,2]}).set_index(['index'])

which gives:

       column
index
a           1
b           2
c           3
a           4
b           1
c           2

Then to get the mean of each subgroup one would:

df.groupby(df.index).mean()

       column
index
a         2.5
b         1.5
c         2.5

However, what I've been trying to achieve without constantly looping and slicing the data, is how do I get the mean for pairs of subgroups?

For instance, the mean of a & b is 2? As if their values were combined.

The output would be something akin to:

       column
index
a & a     2.5
a & b     2.0
a & c     2.5
b & b     1.5
b & c     2.0
c & c     2.5

Preferably this would involve manipulating the parameters in 'groupby' but as it is, I'm having to resort to looping and slicing. With the ability to build all combinations of subgroups at some point.

Answer 1

I've revisited this 3 years later with a general solution to this problem.

It's being used in this open source library, which is why I'm now able to do this here and it works with any number of indexes and creates combinations on them using numpy matrix broadcasting

So first of all, that is not a valid dataframe . The indexes aren't unique. Let's add another index to that object and make it a Series:

df = pd.DataFrame({
    'unique': [1, 2, 3, 4, 5, 6], 
    'index': ['a','b','c','a','b','c'], 
    'column': [1,2,3,4,1,2]
}).set_index(['unique','index'])
s = df['column']

Let's unstack that index:

>>> idxs = ['index'] # set as variable to be used later on
>>> unstacked = s.unstack(idxs)
       column
index       a    b    c
unique
1         1.0  NaN  NaN
2         NaN  2.0  NaN
3         NaN  NaN  3.0
4         4.0  NaN  NaN
5         NaN  1.0  NaN
6         NaN  NaN  2.0
>>> vals = unstacked.values
array([[  1.,  nan,  nan],
       [ nan,   2.,  nan],
       [ nan,  nan,   3.],
       [  4.,  nan,  nan],
       [ nan,   1.,  nan],
       [ nan,  nan,   2.]])

>>> sum = np.nansum(vals, axis=0)
>>> count = (~np.isnan(vals)).sum(axis=0)
>>> mean = (sum + sum[:, np.newaxis]) / (count + count[:, np.newaxis])
array([[ 2.5,  2. ,  2.5],
       [ 2. ,  1.5,  2. ],
       [ 2.5,  2. ,  2.5]])

Now recreate the output dataframe:

>>> new_df = pd.DataFrame(mean, unstacked.columns, unstacked.columns.copy())
index_    a    b    c
index
a       2.5  2.0  2.5
b       2.0  1.5  2.0
c       2.5  2.0  2.5
>>> idxs_ = [ x+'_' for x in idxs ]
>>> new_df.columns.names = idxs_
>>> new_df.stack(idxs_, dropna=False)
index  index_
a      a         2.5
       b         2.0
       c         2.5
b      a         2.0
       b         1.5
       c         2.0
c      a         2.5
       b         2.0
       c         2.5

Answer 2

My current implementation is:

 import pandas as pd
 import itertools
 import numpy as np

    # get all pair of categories here
def all_pairs(df, ix):
    hash = {
        ix: [],
        'p': []
    }
    for subset in itertools.combinations(np.unique(np.array(df.index)), 2):
        hash[ix].append(subset)
        hash['p'].append(df.loc[pd.IndexSlice[subset], :]).mean)

    return pd.DataFrame(hash).set_index(ix)

Which gets the combinations and then adds them to the has that then builds back up into a data frame. It's hacky though :(

Answer 3

Here's an implementation that uses a MultiIndex and an outer join to handle the cross join.

import pandas as pd
from pandas import DataFrame, Series
import numpy as np

df = pd.DataFrame({'index': ['a','b','c','a','b','c'], 'column': [1,2,3,4,1,2]}).set_index(['index'])

groupedDF = df.groupby(df.index).mean()
# Create new MultiIndex using from_product which gives a paring of the elements in each iterable
p = pd.MultiIndex.from_product([groupedDF.index, groupedDF.index])
# Add column for cross join
groupedDF[0] = 0
# Outer Join
groupedDF = pd.merge(groupedDF, groupedDF, how='outer', on=0).set_index(p)
# get mean for every row (which is the average for each pair)
# unstack to get matrix for deduplication
crossJoinMeans = groupedDF[['column_x', 'column_y']].mean(axis=1).unstack()
# Create Identity matrix because each pair of itself will be needed
b = np.identity(3, dtype='bool')
# set the first column to True because it contains the rest of the unique means (the identity portion covers the first part)
b[:,0] = True
# invert for proper use of DataFrame Mask
b = np.invert(b)
finalDF = crossJoinMeans.mask(b).stack()

I'd guess that this could be cleaned up and made more concise.