How can I concatenate two dataframes with different multi-indexes without merging indexes?

Question

I have two dataframes. Each has a two-level multi-index. The first level is the same in each, but the second level is different. I would like to concatenate the dataframes and end up with a dataframe with a three-level multi-index, where records from the first dataframe would have 'NaN' in the third index level, and records from the second dataframe would have 'NaN' in the second index level. Instead, I get a dataframe with a two-level index, where the values in the second level of each dataframe are put in the same index level, which takes the name of the second level in the first dataframe (see code below).

Is there a nice way to do this? I could make the second level of each index into a column, concatenate, then put them back into the index, but this seems like a roundabout way of doing it to me.

df1 = pd.DataFrame({'index-1':['a1','b1','c1','d1'], 'index-2':['a2','b2','c2','d2'], 'values':[1,2,3,4]})
df2 = pd.DataFrame({'index-1':['a1','b1','c1','d1'], 'index-3':['a3','b3','c3','d3'], 'values':[5,6,7,8]})

df1.set_index(['index-1','index-2'], inplace=True)
df2.set_index(['index-1','index-3'], inplace=True)

pd.concat([df1, df2])

Thanks!

Answer 1

It'll be easier to reset the index on the two input dataframes, concat them and then set the index again:

pd.concat([df1.reset_index(), df2.reset_index()], sort=False) \
    .set_index(['index-1', 'index-2', 'index-3'])

Result:

                         values
index-1 index-2 index-3        
a1      a2      NaN           1
b1      b2      NaN           2
c1      c2      NaN           3
d1      d2      NaN           4
a1      NaN     a3            5
b1      NaN     b3            6
c1      NaN     c3            7
d1      NaN     d3            8