熊猫在多索引中连接级别

Question

I do have following excel file:

{0: {0: nan, 1: nan, 2: nan, 3: 'A', 4: 'A', 5: 'B', 6: 'B', 7: 'C', 8: 'C'},
 1: {0: nan, 1: nan, 2: nan, 3: 1.0, 4: 2.0, 5: 1.0, 6: 2.0, 7: 1.0, 8: 2.0},
 2: {0: 'AA1', 1: 'a', 2: 'ng/mL', 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1},
 3: {0: 'AA2', 1: 'a', 2: nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1},
 4: {0: 'BB1', 1: 'b', 2: nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1},
 5: {0: 'BB2', 1: 'b', 2: 'mL', 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1},
 6: {0: 'CC1', 1: 'c', 2: nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1},
 7: {0: 'CC2', 1: 'c', 2: nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}}

I would like to create following dataframe:

level_0     AA1   AA2   CB1  BB2   CC1   CC2
new     a ng/mL a N/A b N/A b mL c N/A c N/A
0 1                                         
A 1           1     1     1    1     1     1
  2           1     1     1    1     1     1
B 1           1     1     1    1     1     1
  2           1     1     1    1     1     1
C 1           1     1     1    1     1     1
  2           1     1     1    1     1     1

What I tried:

# read the column index separately to avoid pandas inputting "Unnamed: ..."
# for the nans
df = pd.read_excel(file_path, skiprows=3, index_col=None, header=None)
df.set_index([0, 1], inplace=True)

# the column index
cols = pd.read_excel(file_path, nrows=3, index_col=None, header=None).loc[:, 2:]
cols = cols.fillna('N/A')
idx = pd.MultiIndex.from_arrays(cols.values)
df.columns = idx

The new dataframe:

      AA1 AA2 CB1 BB2 CC1 CC2
        a   a   b   b   c   c
    ng/mL N/A N/A  mL N/A N/A
0 1                          
A 1     1   1   1   1   1   1
  2     1   1   1   1   1   1
B 1     1   1   1   1   1   1
  2     1   1   1   1   1   1
C 1     1   1   1   1   1   1
  2     1   1   1   1   1   1

This approach works but is kind of tedious:

df1 = df.T.reset_index()
df1['new'] = df1.loc[:, 'level_1'] + ' ' + df1.loc[:, 'level_2']
df1.set_index(['level_0', 'new']).drop(['level_1', 'level_2'], axis=1).T

Which gives me:

level_0     AA1   AA2   CB1  BB2   CC1   CC2
new     a ng/mL a N/A b N/A b mL c N/A c N/A
0 1                                         
A 1           1     1     1    1     1     1
  2           1     1     1    1     1     1
B 1           1     1     1    1     1     1
  2           1     1     1    1     1     1
C 1           1     1     1    1     1     1
  2           1     1     1    1     1     1

Is there a simpler solution available ?

Answer 1

Use:

#file from sample data

d = {0: {0:  np.nan, 1:  np.nan, 2:  np.nan, 3: 'A', 4: 'A', 5: 'B', 6: 'B', 7: 'C', 8: 'C'}, 
     1: {0:  np.nan, 1:  np.nan, 2:  np.nan, 3: 1.0, 4: 2.0, 5: 1.0, 6: 2.0, 7: 1.0, 8: 2.0}, 
     2: {0: 'AA1', 1: 'a', 2: 'ng/mL', 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}, 
     3: {0: 'AA2', 1: 'a', 2:  np.nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}, 
     4: {0: 'BB1', 1: 'b', 2:  np.nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}, 
     5: {0: 'BB2', 1: 'b', 2: 'mL', 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}, 
     6: {0: 'CC1', 1: 'c', 2:  np.nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}, 
     7: {0: 'CC2', 1: 'c', 2:  np.nan, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1}}

df = pd.DataFrame(d)

df.to_excel('file.xlsx', header=False, index=False)

First create MultiIndex DataFrame with header=[0,1,2] , then create MultiIndex by first 2 columns with DataFrame.set_index and remove index names by DataFrame.reset_index :

df = pd.read_excel('file.xlsx', header=[0,1,2])

df = df.set_index(df.columns[:2].tolist()).rename_axis((None, None))

Then loop by each level in list comprehension and join second with third level if not Unnamed , last use MultiIndex.from_tuples :

tuples = [(a, f'{b} N/A') if c.startswith('Unnamed') 
          else (a, f'{b} {c}') 
          for a, b, c in df.columns]

print (tuples)
[('AA1', 'a ng/mL'), ('AA2', 'a N/A'), 
 ('BB1', 'b N/A'), ('BB2', 'b mL'),
 ('CC1', 'c N/A'), ('CC2', 'c N/A')]

df.columns = pd.MultiIndex.from_tuples(tuples)
print (df)
        AA1   AA2   BB1  BB2   CC1   CC2
    a ng/mL a N/A b N/A b mL c N/A c N/A
A 1       1     1     1    1     1     1
  2       1     1     1    1     1     1
B 1       1     1     1    1     1     1
  2       1     1     1    1     1     1
C 1       1     1     1    1     1     1
  2       1     1     1    1     1     1

Another idea is use:

df = pd.read_excel('file.xlsx', header=[0,1,2])
df = df.set_index(df.columns[:2].tolist()).rename_axis((None, None))

lv1 = df.columns.get_level_values(0)
lv2 = df.columns.get_level_values(1)
lv3 = df.columns.get_level_values(2)
lv3 = lv3.where(~lv3.str.startswith('Unnamed'),'N/A')

df.columns = [lv1, lv2.to_series() + ' ' + lv3]