Merge two column header and give a new name in MultiIndex Dataframe python/Add column above column names

Question

I have the initial dataframe:

              r_id1   r_score1    rid2     r_score2
Rank
ID1    ID2
1      A-1    id-1      1.23      id-34      6.78
2      A-1    id-9      2.34      id-45      3.45
3      A-2    id-8      3.56      id-32      4.56
4      A-3    id-6      4.35      id-10      3.98
5      A-4    id-4      7.89      id-67      2.98

I want my data frame to be (Result_df):

                 Score_R1              Score_R2
             r_id1   r_score1     rid2     r_score2
ID1    ID2
1      A-1    id-1      1.23      id-34      6.78
2      A-1    id-9      2.34      id-45      3.45
3      A-2    id-8      3.56      id-32      4.56
4      A-3    id-6      4.35      id-10      3.98
5      A-4    id-4      7.89      id-67      2.98

My dataframe is multiindex and with MultiIndex columns. I tried this piece of code

final_df.columns = [' '.join(col).strip() for col in final_df.columns.values]

which gives me this output

ID1    ID2    r_id1   r_score1    rid2     r_score2
1      A-1    id-1      1.23      id-34      6.78
2      A-1    id-9      2.34      id-45      3.45
3      A-2    id-8      3.56      id-32      4.56
4      A-3    id-6      4.35      id-10      3.98
5      A-4    id-4      7.89      id-67      2.98

After:

cols = final_df.columns.map(''.join)

lvl = 'Score_R' + cols.str.extract('(\d+)', expand=False)

final_df.columns = [lvl, cols]

final_df.to_csv("f.csv")

Output is:

             Score_R1  Score_R1    Score_R2  Score_R2
               r_id1   r_score1     rid2     r_score2
ID1    ID2
1      A-1    id-1      1.23      id-34      6.78
2      A-1    id-9      2.34      id-45      3.45
3      A-2    id-8      3.56      id-32      4.56
4      A-3    id-6      4.35      id-10      3.98
5      A-4    id-4      7.89      id-67      2.98

I need to combine the column headers with same name

                Score_R1               Score_R2  
              r_id1   r_score1     rid2     r_score2
ID1    ID2
1      A-1    id-1      1.23      id-34      6.78
2      A-1    id-9      2.34      id-45      3.45

Answer 1

You can use str.extract for get numbers from columns names, add prefix and last assign back with original columns for MultiIndex in columns :

print (df.columns.tolist())
[('r_id1', ''), ('r_score1', ''), ('rid2', ''), ('r_score2', '')]

cols = df.columns.map(''.join)
print (cols.tolist())
['r_id1', 'r_score1', 'rid2', 'r_score2']

lvl = 'Score_R' + cols.str.extract('(\d+)', expand=False)
print (lvl)
Index(['Score_R1', 'Score_R1', 'Score_R2', 'Score_R2'], dtype='object')


df.columns = [lvl, cols]
print (df)
        Score_R1          Score_R2         
           r_id1 r_score1     rid2 r_score2
ID1 ID2                                    
1   A-1     id-1     1.23    id-34     6.78
2   A-1     id-9     2.34    id-45     3.45
3   A-2     id-8     3.56    id-32     4.56
4   A-3     id-6     4.35    id-10     3.98
5   A-4     id-4     7.89    id-67     2.98


df.columns = df.columns.map('_'.join)
print (df)
        Score_R1_r_id1  Score_R1_r_score1 Score_R2_rid2  Score_R2_r_score2
ID1 ID2                                                                   
1   A-1           id-1               1.23         id-34               6.78
2   A-1           id-9               2.34         id-45               3.45
3   A-2           id-8               3.56         id-32               4.56
4   A-3           id-6               4.35         id-10               3.98
5   A-4           id-4               7.89         id-67               2.98

EDIT: You can replace missing values of first level to empty strings:

cols = df.columns.droplevel(-1)
lvl = 'Score_R' + cols.str.extract('(\d+)', expand=False)
print (lvl)
Index(['Score_R1', 'Score_R1', 'Score_R2', 'Score_R2'], dtype='object')

lvl = lvl.where(~lvl.duplicated(), '')
print (lvl)
Index(['Score_R1', '', 'Score_R2', ''], dtype='object')

df.columns = [lvl, cols]
print (df)
        Score_R1          Score_R2         
           r_id1 r_score1     rid2 r_score2
ID1 ID2                                    
1   A-1     id-1     1.23    id-34     6.78
2   A-1     id-9     2.34    id-45     3.45
3   A-2     id-8     3.56    id-32     4.56
4   A-3     id-6     4.35    id-10     3.98
5   A-4     id-4     7.89    id-67     2.98

print (df.columns)
MultiIndex([('Score_R1',    'r_id1'),
            (        '', 'r_score1'),
            ('Score_R2',     'rid2'),
            (        '', 'r_score2')],
           )