pandas Add new "rank" columns for every column

Question

I have a df like so (actual df has 4.5 mil rows, 23 cols):

group  feature  col1  col2  col3
g1     f1       1     10    100
g1     f1       11    9     1000
g1     f2       0     8     200
g2     f1       2     7     330
g2     f2       3     7     331
g2     f3       1     7     100
g3     f1       1     6     101
g3     f1       5     9     100
g3     f1       1     8     100

I want to add two new "rank" cols for each col in my df. I will evaluate different cols differently, such as sum, mean, max, etc. For ease of explanation I've broken the problem out into two separate problems below.

I have been advised here to use .loc and not use groupby , but any solution that works is fine. I've tried both and had little success (see here )

The first rank col will rank each feature on the values in col1, col2, and col3 within each group .

At an intermediate stage it would look something like this:

group  feature  col1  col1_sum  col1_rank  col2  col2_avg  col2_rank  col3 col3_max  col3_rank
g1     f1       1     12        1          10    9.5       1          100  1000      1
g1     f1       11                         9                          1000           
g1     f2       0     0         2          8     8         2          200  200       2
g2     f1       2     2         2          7     7         1          330  330       2
g2     f2       3     3         1          7     7         1          331  331       1
g2     f3       1     1         3          7     7         1          100  100       3
g3     f1       1     7         1          6     7.67      1          101  101       1
g3     f1       5                          9                          100            
g3     f1       1                          8                          100            

It will output this:

group  feature  col1_rank  col2_rank  col3_rank
g1     f1       1          1          1
g1     f2       2          2          2
g2     f1       2          1          2
g2     f2       1          1          1
g2     f3       3          1          3
g3     f1       1          1          1

The second rank col will rank each group by feature on the values in col1, col2, and col3 against all other groups .

At an intermediate stage it would look something like this:

group  feature  col1  col1_sum  col1_rank  col2  col2_avg  col2_rank  col3 col3_max  col3_rank
g1     f1       1     12        1          10    9.5       1          100  1000      1
g1     f1       11                         9                          1000           
g2     f1       2     2         3          7     7         3          330  330       2
g3     f1       1     7         2          6     7.67      2          101  101       3
g3     f1       5                          9                          100            
g3     f1       1                          8                          100            

g1     f2       0     0         2          8     8         1          200  200       2
g2     f2       3     3         1          7     7         2          331  331       1

g2     f3       1     1         1          7     7         1          100  100       1

It will output this:

group  feature  col1_rank  col2_rank  col3_rank
g1     f1       1          1          1
g2     f1       3          3          2
g3     f1       2          2          3
g1     f2       2          1          2
g2     f2       1          2          1
g2     f3       1          1          1

Answer 1

I would use groupby on ['group', 'feature'] to produce an intermediary dataframe containing the sum, avg and max columns (not the ranks), and then again groupby on group only to produce the ranks.

Intermediary dataframe:

df2 = pd.concat([
    df.iloc[:,[0,1,2]].groupby(['group', 'feature']).sum(),
    df.iloc[:,[0,1,3]].groupby(['group', 'feature']).mean(),
    df.iloc[:,[0,1,4]].groupby(['group', 'feature']).max()
    ], axis=1)

The intermediary dataframe is:

               col1      col2  col3
group feature                      
g1    f1         12  9.500000  1000
      f2          0  8.000000   200
g2    f1          2  7.000000   330
      f2          3  7.000000   331
      f3          1  7.000000   100
g3    f1          7  7.666667   101

Now for the final dataframe:

df3 = df2.groupby('group').rank(method='min', ascending=False).reset_index()

which finally gives:

  group feature  col1  col2  col3
0    g1      f1   1.0   1.0   1.0
1    g1      f2   2.0   2.0   2.0
2    g2      f1   2.0   1.0   2.0
3    g2      f2   1.0   1.0   1.0
4    g2      f3   3.0   1.0   3.0
5    g3      f1   1.0   1.0   1.0

---

For the second part of the question, I would just change the indexing of the intermediary dataframe, and compute ranks after grouping on 'feature' :

dfx4 = dfx.reset_index().set_index(['feature', 'group']
                                   ).sort_index().groupby('feature').rank(
                                   method='min', ascending=False
                                   ).reset_index()

which gives:

  feature group  col1  col2  col3
0      f1    g1   1.0   1.0   1.0
1      f1    g2   3.0   3.0   2.0
2      f1    g3   2.0   2.0   3.0
3      f2    g1   2.0   1.0   2.0
4      f2    g2   1.0   2.0   1.0
5      f3    g2   1.0   1.0   1.0