根据另一列的名称重命名Pandas Multiindex

Question

I've got a CSV file that is generated in a format I cannot change. The file has a multiindex: headers on two lines. The first line (higher level of index) has blanks when the value doesn't change.

What my header looks like:

What it actually comes down to and what I want:

I would like to be able to process it correctly in Python 2.7 with Pandas.

I resulted to looping on the first level of index and if the value is blank, set it to be the same as the one on the left.

I start by loading the dataframe in pandas:

df = pd.read_csv(myFile, header=[0,1], sep=',')
df

I've tried the following:

for i, val in enumerate(df.columns.values):
    if val[0][:7] == 'Unnamed':
        l.append([l[i-1][0], val[1]])
    else:
        l.append(val)

The list "l" I'm getting appears to be what I want:

[('Foo', 'A'),
 ['Foo', 'B'],
 ['Foo', 'C'],
 ('Bar', 'A'),
 ['Bar', 'B'],
 ['Bar', 'C']]

I've tried both:

df.columns = l 

Produces a non multiindex dataframe

index = pd.MultiIndex.from_tuples(l)
df.reindex(columns = index)

This one gives me the correct index, but values disappear.

I'm getting a strong gut feeling that the entire approach I'm trying isn't very pythonic nor does it make sense to use a list then converted to a dict. Any idea how I can multiindex properly?

Answer 1

Instead of using reindex , set the columns to your new index directly:

df.columns = pd.MultiIndex.from_tuples(l)

That should produce the desired result.

reindex doesn't just replace the index values (though that sounds like what it should do, and the documentation isn't especially clear). Instead it goes through your new indices, picks the rows or columns that match the new indices, and puts NaN where no old index matches a new index. That's what's happening to you: when reindex hits ['Foo', 'B'] , which doesn't exist in your original dataframe, it fills the column in the new dataframe with NaN .

If your columns are always going to follow a consistent pattern (one top-level column for every three second-level columns, for example), you could also use MultiIndex.from_product to make the column index:

iterables = [["Foo", "Bar"], ["A", "B", "C"]]
index = pd.MultiIndex.from_product(iterables)