How do I apply transformations to list of pandas dataframes?

Question

I have a bunch of pandas DataFrames belonging to a handful of logical groupings, but all of which have some overlapping columns. and it would save a lot of time if I could apply a list of functions (like the one in funcs below) to a whole list of DataFrames.

# Make example DataFrames
df_a = pd.DataFrame({'col_a': [1, 1, 2], 'col_b': [1, 1, 2], 'col_c': [1, 1, 2], 
                     'col_d': [1, 2, 3], 'col_e': [1, 2, 3], 'col_f': [1, 2, 3], 
                     'foo': 'foo', 'bar': 'bar', 'baz': 'baz'})
df_b = pd.DataFrame({'col_a': [4, 5, 5], 'col_b': [4, 5, 5], 'col_c': [4, 5, 5], 
                     'col_d': [4, 5, 6], 'col_e': [4, 5, 6], 'col_f': [4, 5, 6],
                     'foo': 'foo', 'bar': 'bar', 'baz': 'baz'})
df_c = pd.DataFrame({'col_a': [7, 7, 7], 'col_b': [7, 7, 7], 'col_c': [7, 7, 7], 
                     'col_d': [7, 8, 9], 'col_e': [7, 8, 9], 'col_f': [7, 8, 9], 
                     'foo': 'foo', 'bar': 'bar', 'baz': 'baz'})

# Make list of a bunch of DataFrames
data_sets_a = [df_a, df_b, df_c]

# Drop some columns (this works as expected on each DataFrame)
[d.drop(['foo', 'bar', 'baz'], axis=1, inplace=True) for d in data_sets_a]

# List of functions to apply to overlapping DataFrame columns
funcs = {'col_d': 'count', 'col_e': 'min', 'col_f': 'sum'}

# Group by and aggregate with funcs dict (does not work)
[d.groupby(['col_a', 'col_b', 'col_c']).agg(funcs, inplace=True).reset_index() for d in data_sets_a]

data_sets_a

Using drop with inplace=True over a list of DataFrames in a list comprehension works as I expected, but it doesn't work with groupby and agg --the DataFrames in the list remain unchanged.

[   col_a  col_b  col_c  col_d  col_e  col_f
 0      1      1      1      1      1      1
 1      1      1      1      2      2      2
 2      2      2      2      3      3      3,
    col_a  col_b  col_c  col_d  col_e  col_f
 0      4      4      4      4      4      4
 1      5      5      5      5      5      5
 2      5      5      5      6      6      6,
    col_a  col_b  col_c  col_d  col_e  col_f
 0      7      7      7      7      7      7
 1      7      7      7      8      8      8
 2      7      7      7      9      9      9]

Changing the inplace=True value for drop does what I'd expect, but it doesn't seem to make a difference with groupby and agg .

Can someone explain why the two list comprehensions have different results, or show me a better way to get the results I'm looking for?

Is it a problem with the code mapping the functions to the DataFrame list?

I've been reading pandas' documentation and Googling for a while now and tried various things like query , map , lambda combinations, but to no avail.

Answer 1

Solution

for i in range(len(data_sets_a)):
    cols = ['col_a', 'col_b', 'col_c']
    gb = data_sets_a[i].groupby(cols)
    data_sets_a[i] = gb.agg(funcs, inplace=1).reset_index()

Explanation

If your list comprehension, you were returning the correct objects but not placing them where you wanted. The inplace=True was not augmenting the same object being pointed to in the list data_sets_a .

What I did was to assign to each element of the list the correct augmentation.

Another way to have done it is to use what you already had:

data_sets_a = [
    d.groupby(
        ['col_a', 'col_b', 'col_c']
    ).agg(funcs, inplace=True).reset_index() for d in data_sets_a
]

just assign the new list to the old list.

Answer 2

If I understand your question correctly, the problem is with your funcs . You can try it this way instead:

def funcs(x):
    col_d = x['col_d'].count()
    col_e = x['col_e'].min()
    col_f = x['col_f'].sum()
    return pd.Series([col_d, col_e, col_f], index= ['col_d', 'col_e', 'col_f'] )

Then you can use apply(funcs)

[d.groupby(['col_a', 'col_b', 'col_c']).apply(funcs).reset_index() for d in data_sets_a]

The output will be:

[   col_a  col_b  col_c  col_d  col_e  col_f
 0      1      1      1      2      1      3
 1      2      2      2      1      3      3,
    col_a  col_b  col_c  col_d  col_e  col_f
 0      4      4      4      1      4      4
 1      5      5      5      2      5     11,
    col_a  col_b  col_c  col_d  col_e  col_f
 0      7      7      7      3      7     24]