如何在熊猫中筛选分组依据

Question

I am a newbie in Pandas. I have the following dataset. Think of the dataset as departments('k1') and people('k2') of a company.

dframe = pd.DataFrame({'k1': ['X','X','Y','Y','Z','Z'],
   ...:                         'k2': ['P1','P2','P3','P4','P5','P6'],
   ...:                          'dataset1': np.random.randn(6)})
   ...:

If I take the mean grouped by dept/'k1' I get the following

   dataset1
k1
X   0.153825
Y  -0.648500
Z   1.133334

If I take the mean grouped by people/'k2', I get the following

In [6]: dframe.groupby('k2').mean()
Out[6]:
    dataset1
k2
P1  1.595455
P2 -1.287805
P3  0.211858
P4 -1.508859
P5  1.350336
P6  0.916332

My question is how can I filter only the mean values grouped by people/'k2' which is greater than the mean of dept/'k1' to which it belongs. eg P1 mean value is greater than X mean value to which it belongs

Out[6]:
    dataset1
k2
P1  1.595455
P3  0.211858
P5  1.350336

Answer 1

Sample (changed P6 to P5 ):

np.random.seed(45)
dframe = pd.DataFrame({'k1': ['X','X','Y','Y','Z','Z'],
                         'k2': ['P1','P2','P3','P4','P5','P5'],
                            'dataset1': np.random.randn(6)})

print (dframe)
   dataset1 k1  k2
0  0.026375  X  P1
1  0.260322  X  P2
2 -0.395146  Y  P3
3 -0.204301  Y  P4
4 -1.271633  Z  P5
5 -2.596879  Z  P5

First create new column by groupby and transform :

dframe['meank1'] = dframe.groupby('k1').transform('mean')
print (dframe)
   dataset1 k1  k2    meank1
0  0.026375  X  P1  0.143348
1  0.260322  X  P2  0.143348
2 -0.395146  Y  P3 -0.299723
3 -0.204301  Y  P4 -0.299723
4 -1.271633  Z  P5 -1.934256
5 -2.596879  Z  P5 -1.934256

Then aggregate by agg mean and first , also is necessary add k1 column to groupby for avoid wrong output if same k2 in another k1 .

dframe = dframe.groupby(['k1','k2']).agg({'dataset1':'mean', 'meank1':'first'})
print (dframe)
         meank1  dataset1
k1 k2                    
X  P1  0.143348  0.026375
   P2  0.143348  0.260322
Y  P3 -0.299723 -0.395146
   P4 -0.299723 -0.204301
Z  P5 -1.934256 -1.934256

Last filter by boolean indexing or query :

dframe = dframe.loc[dframe['meank1'] > dframe['dataset1'], ['dataset1']]
#alternative sol
#dframe = dframe.query('meank1 > dataset1')[['dataset1']]
print (dframe)
       dataset1
k1 k2          
X  P1  0.026375
Y  P3 -0.395146

And if want remove first level of MultiIndex add reset_index :

dframe = dframe.reset_index(level=0, drop=True)
print (dframe)
    dataset1
k2          
P1  0.026375
P3 -0.395146

For column from index use:

dframe = dframe.reset_index(level=0, drop=True).reset_index()
print (dframe)
   k2  dataset1
0  P1  0.026375
1  P3 -0.395146