有没有更好的方法来聚合同一分组 pandas dataframe 上的多个列？

Question

I am trying to figure out how should I manipulate my data so I can aggregate on multiple columns but for same grouped pandas data. The reason why I am doing this because, I need to get stacked line chart which take data from different aggregation on same grouped data. How can we do this some compact way? can anyone suggest possible way of doing this in pandas? any ideas?

my current attempt :

import pandas as pd
import matplotlib.pyplot as plt

url = "https://gist.githubusercontent.com/adamFlyn/4657714653398e9269263a7c8ad4bb8a/raw/fa6709a0c41888503509e569ace63606d2e5c2ff/mydf.csv"
df = pd.read_csv(url, parse_dates=['date'])

df_re = df[df['retail_item'].str.contains("GROUND BEEF")]
df_rei = df_re.groupby(['date', 'retail_item']).agg({'number_of_ads': 'sum'})
df_rei = df_rei.reset_index(level=[0,1])
df_rei['week'] = pd.DatetimeIndex(df_rei['date']).week
df_rei['year'] = pd.DatetimeIndex(df_rei['date']).year
df_rei['week'] = df_rei['date'].dt.strftime('%W').astype('uint8')

df_ret_df1 = df_rei.groupby(['retail_item', 'week'])['number_of_ads'].agg([max, min, 'mean']).stack().reset_index(level=[2]).rename(columns={'level_2': 'mm', 0: 'vals'}).reset_index()

similarly, I need to do data aggregation also like this:

df_re['price_gap'] = df_re['high_price'] - df_re['low_price']
dff_rei1 = df_re.groupby(['date', 'retail_item']).agg({'price_gap': 'mean'})
dff_rei1 = dff_rei1.reset_index(level=[0,1])
dff_rei1['week'] = pd.DatetimeIndex(dff_rei1['date']).week
dff_rei1['year'] = pd.DatetimeIndex(dff_rei1['date']).year
dff_rei1['week'] = dff_rei1['date'].dt.strftime('%W').astype('uint8')

dff_ret_df2 = dff_rei1.groupby(['retail_item', 'week'])['price_gap'].agg([max, min, 'mean']).stack().reset_index(level=[2]).rename(columns={'level_2': 'mm', 0: 'vals'}).reset_index()

problem

when I made data aggregation, those lines are similar:

df_rei = df_re.groupby(['date', 'retail_item']).agg({'number_of_ads': 'sum'})
df_ret_df1 = df_rei.groupby(['retail_item', 'week'])['number_of_ads'].agg([max, min, 'mean']).stack().reset_index(level=[2]).rename(columns={'level_2': 'mm', 0: 'vals'}).reset_index()

and

dff_rei1 = df_re.groupby(['date', 'retail_item']).agg({'price_gap': 'mean'})
 dff_ret_df2 = dff_rei1.groupby(['retail_item', 'week'])['price_gap'].agg([max, min, 'mean']).stack().reset_index(level=[2]).rename(columns={'level_2': 'mm', 0: 'vals'}).reset_index()

I think better way could be I have to make custom function with *arg , **kwargs to make shift for aggregating the columns, but how should I show stacked line chart where y axis shows different quantities. Is that doable to do so in pandas ?

line plot

I did for getting line chart as follow:

for g, d in df_ret_df1.groupby('retail_item'):
    fig, ax = plt.subplots(figsize=(7, 4), dpi=144)
    sns.lineplot(x='week', y='vals', hue='mm', data=d,alpha=.8)
    y1 = d[d.mm == 'max']
    y2 = d[d.mm == 'min']
    plt.fill_between(x=y1.week, y1=y1.vals, y2=y2.vals)
    
    for year in df['year'].unique():
        data = df_rei[(df_rei.date.dt.year == year) & (df_rei.retail_item == g)]
        sns.lineplot(x='week', y='price_gap', ci=None, data=data, palette=cmap,label=year,alpha=.8)

I want to minimize those so I could able to aggregate on different columns and make stacked line chart, where they share x-axis as week, and y axis shows number of ads and price_range respectively. I don't know is there any better way of doing this. I am doing this because stacked line chart (two vertical subplots), one shows number of ads on y axis and another one shows price ranges for same items along 52 weeks. can anyone suggest any possible way of doing this? any ideas?

Answer 1

This answer builds on the one by Andreas who has already answered the main question of how to produce aggregate variables of multiple columns in a compact way. The goal here is to implement that solution specifically to your case and to give an example of how to produce a single figure from the aggregated data. Here are some key points:

• The dates in the original dataset are already on a weekly frequency so groupby('week') is not needed for df_ret_df1 and dff_ret_df2 , which is why these contain identical values for min, max, and mean.
• This example uses pandas and matplotlib so the variables do not need to be stacked as when using seaborn.
• The aggregation step produces a MultiIndex for the columns. You can access the aggregated variables (min, max, mean) of each high-level variable by using df.xs .
• The date is set as the index of the aggregated dataframe to use as the x variable. Using the DatetimeIndex as the x variable gives you more flexibility for formatting the tick labels and ensures that the data is always plotted in chronological order.
• It is not clear in the question how the data for separate years should be displayed (in separate figures?) so here the entire time series is shown in a single figure.

Import dataset and aggregate it as needed

import pandas as pd              # v 1.2.3
import matplotlib.pyplot as plt  # v 3.3.4

# Import dataset
url = 'https://gist.githubusercontent.com/adamFlyn/4657714653398e9269263a7c8ad4bb8a/\
raw/fa6709a0c41888503509e569ace63606d2e5c2ff/mydf.csv'
df = pd.read_csv(url, parse_dates=['date'])

# Create dataframe containing data for ground beef products, compute
# aggregate variables, and set the date as the index
df_gbeef = df[df['retail_item'].str.contains('GROUND BEEF')].copy()
df_gbeef['price_gap'] = df_gbeef['high_price'] - df_gbeef['low_price']
agg_dict = {'number_of_ads': [min, max, 'mean'],
            'price_gap': [min, max, 'mean']}
df_gbeef_agg = (df_gbeef.groupby(['date', 'retail_item']).agg(agg_dict)
                .reset_index('retail_item'))
df_gbeef_agg

Plot aggregated variables in single figure containing small multiples

variables = ['number_of_ads', 'price_gap']
colors = ['tab:orange', 'tab:blue']
nrows = len(variables)
ncols = df_gbeef_agg['retail_item'].nunique()

fig, axs = plt.subplots(nrows, ncols, figsize=(10, 5), sharex=True, sharey='row')
for axs_row, var, color in zip(axs, variables, colors):
    for i, (item, df_item) in enumerate(df_gbeef_agg.groupby('retail_item')):
        ax = axs_row[i]
        
        # Select data and plot it
        data = df_item.xs(var, axis=1)
        ax.fill_between(x=data.index, y1=data['min'], y2=data['max'],
                        color=color, alpha=0.3, label='min/max')
        ax.plot(data.index, data['mean'], color=color, label='mean')
        ax.spines['bottom'].set_position('zero')
        
        # Format x-axis tick labels
        fmt = plt.matplotlib.dates.DateFormatter('%W') # is not equal to ISO week
        ax.xaxis.set_major_formatter(fmt)
        
        # Fomat subplot according to position within the figure
        if ax.is_first_row():
            ax.set_title(item, pad=10)
        if ax.is_last_row():
            ax.set_xlabel('Week number', size=12, labelpad=5)
        if ax.is_first_col():
            ax.set_ylabel(var, size=12, labelpad=10)
        if ax.is_last_col():
            ax.legend(frameon=False)

fig.suptitle('Cross-regional weekly ads and price gaps of ground beef products',
             size=14, y=1.02)
fig.subplots_adjust(hspace=0.1);

Answer 2

I am not sure if this answers your question fully, but based on your headline I guess it all boils down to:

import pandas as pd

url = "https://gist.githubusercontent.com/adamFlyn/4657714653398e9269263a7c8ad4bb8a/raw/fa6709a0c41888503509e569ace63606d2e5c2ff/mydf.csv"
df = pd.read_csv(url, parse_dates=['date'])

# define which columns to group and in which way
dct = {'low_price': [max, min],
       'high_price': min,
       'year': 'mean'}

# actually group the columns
df.groupby(['region']).agg(dct)

Output:

              low_price       high_price         year
                    max   min        min         mean
region
ALASKA            16.99  1.33       1.33  2020.792123
HAWAII            12.99  1.33       1.33  2020.738318
MIDWEST           28.73  0.99       0.99  2020.690159
NORTHEAST         19.99  1.20       1.99  2020.709916
NORTHWEST         16.99  1.33       1.33  2020.736397
SOUTH CENTRAL     28.76  1.20       1.49  2020.700980
SOUTHEAST         21.99  1.33       1.48  2020.699655
SOUTHWEST         16.99  1.29       1.29  2020.704341