Mean of a grouped-by pandas dataframe with flexible aggregation period

Question

As here I need to calculate the mean of the colums duration and km for the rows with value ==1 and values = 0. This time I would like that the aggregation period is flexible.

df
Out[20]: 
                          Date duration km   value
0   2015-03-28 09:07:00.800001    0      0    0
1   2015-03-28 09:36:01.819998    1      2    1
2   2015-03-30 09:36:06.839997    1      3    1 
3   2015-03-30 09:37:27.659997    nan    5    0 
4   2015-04-22 09:51:40.440003    3      7    0
5   2015-04-23 10:15:25.080002    0      nan  1

For the aggregation period of 1 day I can use the solution suggested before:

df.pivot_table(values=['duration','km'],columns=['value'],index=df['Date'].dt.date,aggfunc='mean'

ndf.columns = [i[0]+str(i[1]) for i in ndf.columns]

            duration0  duration1  km0  km1
Date                                      
2015-03-28        0.0        1.0  0.0  2.0
2015-03-30        NaN        1.0  5.0  3.0
2015-04-22        3.0        NaN  7.0  NaN
2015-04-23        NaN        0.0  NaN  NaN

However, I do not know how to change the aggregation period in case, for example, I want to pass it as an argument of a function... For this reason an approach with pd.Grouper(freq=freq_aggregation) , being freq_aggregation = 'd' or '60s' would be preferred...

Answer 1

Let's use pd.Grouper , unstack , and columns map:

freq_str = '60s'
df_out = df.groupby([pd.Grouper(freq=freq_str, key='Date'),'value'])['duration','km'].agg('mean').unstack()

df_out.columns = df_out.columns.map('{0[0]}{0[1]}'.format)

df_out

Output:

                     duration0  duration1  km0  km1
Date                                               
2015-03-28 09:07:00        0.0        NaN  0.0  NaN
2015-03-28 09:36:00        NaN        1.0  NaN  2.0
2015-03-30 09:36:00        NaN        1.0  NaN  3.0
2015-03-30 09:37:00        NaN        NaN  5.0  NaN
2015-04-22 09:51:00        3.0        NaN  7.0  NaN
2015-04-23 10:15:00        NaN        0.0  NaN  NaN

Now, let's change freq_str to 'D':

freq_str = 'D'
df_out = df.groupby([pd.Grouper(freq=freq_str, key='Date'),'value'])['duration','km'].agg('mean').unstack()

df_out.columns = df_out.columns.map('{0[0]}{0[1]}'.format)

print(df_out)

Output:

            duration0  duration1  km0  km1
Date                                      
2015-03-28        0.0        1.0  0.0  2.0
2015-03-30        NaN        1.0  5.0  3.0
2015-04-22        3.0        NaN  7.0  NaN
2015-04-23        NaN        0.0  NaN  NaN

Answer 2

You can pass grouper to the index of pivot table. Hope this is what you are looking for ie

ndf = df.pivot_table(values=['duration','km'],columns=['value'],index=pd.Grouper(key='Date', freq='60s'),aggfunc='mean')
ndf.columns = [i[0]+str(i[1]) for i in ndf.columns]

Output:

duration0  duration1  km0  km1
Date                                               
2015-03-28 09:07:00        0.0        NaN  0.0  NaN
2015-03-28 09:36:00        NaN        1.0  NaN  2.0
2015-03-30 09:36:00        NaN        1.0  NaN  3.0
2015-03-30 09:37:00        NaN        NaN  5.0  NaN
2015-04-22 09:51:00        3.0        NaN  7.0  NaN
2015-04-23 10:15:00        NaN        0.0  NaN  NaN

If frequency is D then

duration0  duration1  km0  km1
Date                                      
2015-03-28        0.0        1.0  0.0  2.0
2015-03-30        NaN        1.0  5.0  3.0
2015-04-22        3.0        NaN  7.0  NaN
2015-04-23        NaN        0.0  NaN  NaN

Answer 3

use groupby

df = df.set_index('Date')    
df.groupby([pd.TimeGrouper('D'), 'value']).mean()

                 duration   km
Date       value               
2017-10-11 0      1.500000  4.0
           1      0.666667  2.5


df.groupby([pd.TimeGrouper('60s'), 'value']).mean()

                           duration   km
Date                value               
2017-10-11 09:07:00 0      0.0       0.0
2017-10-11 09:36:00 1      1.0       2.5
2017-10-11 09:37:00 0     NaN        5.0
2017-10-11 09:51:00 0      3.0       7.0
2017-10-11 10:15:00 1      0.0      NaN 

if you want it unstacked, then unstack it.

df.groupby([pd.TimeGrouper('D'), 'value']).mean().unstack()

           duration        km     
value             0    1    0    1
Date                              
2017-10-11 1.50     0.67 4.00 2.50