python pandas dataframe groupby or pivot_table

Question

Example:

import pandas as pd
data = {'id':[101,101,101,101,102,102,102,102],
    'day':[1,2,1,2,1,2,1,2],
    'year':[2011,2011,2012,2012,2011,2011,2012,2012],
    'avg':[0.500,0.400,0.300,0.200,0.555,0.455,0.355,0.255],
    'sum':[1, 2, 2, 3, 6, 6, 8, 9],
    'div':[2, 1, 3, 2, 6, 1, 6, 3]}
df = pd.DataFrame(data)
df

    id  day year    avg     sum div
0   101 1   2011    0.500   1   2
1   101 2   2011    0.400   2   1
2   101 1   2012    0.300   2   3
3   101 2   2012    0.200   3   2
4   102 1   2011    0.555   6   6
5   102 2   2011    0.455   6   1
6   102 1   2012    0.355   8   6
7   102 2   2012    0.255   9   3

desired output:

    id  sum div 2011_avg    2012_avg    2011_sum    2012_sum    2011_div    2012_div
0   101 8   8   0.450       0.250       3           5           2           1.5
1   102 29  16  0.505       0.305       12          17          6           2.0

I made several pivot_tables for each column by year and joins multiple times..

Can anyone give me some knowledge to a easier or efficient way to get desired output?

Answer 1

You may need groupby two times , then join the result back

s=df.groupby(['id','year']).agg({'avg':'mean','sum':'sum','div':lambda x : x.iloc[0]/x.iloc[1]})
s=s.unstack()# here is reshape 
s.columns=s.columns.map('{0[1]}_{0[0]}'.format) # here is flatten the multiple index 
s
Out[723]:
     2011_avg  2012_avg  2011_sum  2012_sum  2011_div  2012_div
id
101     0.450     0.250         3         5       2.0       1.5
102     0.505     0.305        12        17       6.0       2.0

s2=df.groupby(['id']).agg({'sum':'sum','div':lambda x : x.iloc[0]/x.iloc[1]})

Finaldf=s2.join(s)# join back 

Finaldf
Out[729]: 
     sum  div  2011_avg    ...     2012_sum  2011_div  2012_div
id                         ...                                 
101    8    2     0.450    ...            5       2.0       1.5
102   29    6     0.505    ...           17       6.0       2.0
[2 rows x 8 columns]

Answer 2

I tried just doing 2 groupbys and then merging both results. Just for naming conventions I used the pivot_table.

df2 = df.groupby(by=["id","year"]).agg({
    "avg": np.median,
    "sum": np.sum,
    "div": lambda x : x.iloc[0]/x.iloc[1]
}).reset_index().pivot_table(values=["avg","sum","div"],columns=["year"],index=["id"]).reset_index()
df2.columns = [str(col[1])+"_"+col[0] if col[1] != "" else col[0] for col in df2.columns.values ]
df2.merge(df.groupby(by=["id"]).agg({
    "avg": np.median,
    "div": lambda x : x.iloc[0]/x.iloc[1],
    "sum": np.sum
}), on="id")