Iterating through MultiIndex data in python pandas

Question

I want to be able to iterate through a pandas DataFrame with grouping on a multi-index. Here, I'd like to be able to process a group of rows in each industry all together. I load with a multi-index.

from StringIO import StringIO
data = """industry,location,number
retail,brazil,294
technology,china,100
retail,nyc,2913
retail,paris,382
technology,us,2182
"""

df = pd.read_csv(StringIO(data), sep=",", index_col=['industry', 'location'])

So I wish there was something to this effect:

for industry, rows in df.iter_multiindex():
    for row in rows:
        process_row(row)

Is there such a way to do this?

Answer 1

You can groupby the first level of the multi-index (the industries), and then iterate trough the groups:

In [102]: for name, group in df.groupby(level='industry'):
   .....:     print name, '\n', group, '\n'
   .....:
retail
                   number
industry location
retail   brazil       294
         nyc         2913
         paris        382

technology
                     number
industry   location
technology china        100
           us          2182

group will be each time a dataframe, and you can then iterate through that (with eg for row in group.iterrows() .

But , in most cases such iteration is not needed! What would process_row entail? Probably you can do this in a vectorized manner, directly on the groupby object.

Answer 2

not sure why do you want to do this, but you can do it like this:

for x in df.index:
    print x[0] # industry
    process(df.loc[x]) # row

But it's not how you usually work with DataFrame, you probably want to read about apply() ( Essential Basic Functionality is also really helpful)