通过在字典中附加熊猫数据框来创建字典

Question

I have a dictionary of pandas DataFrames. I want to combine the dataframes in this dictionary to create a dictionary of fewer keys but with larger dataframes as values.

For example, in the example below, I want create d_new from d .

d = {1: pd.DataFrame({'a':[1,2],'b':[3,4]}), 
     2: pd.DataFrame({'a':[3,4],'b':[5,6]}),
     3: pd.DataFrame({'a':[10,11],'b':[12,13]}), 
     4: pd.DataFrame({'a':[12,13],'b':[14,15]})}

d_new = {1:pd.DataFrame({'a':[1,2,3,4],'b':[3,4,5,6]}), 
         3:pd.DataFrame({'a':[10,11,12,13],'b':[12,13,14,15]})}

I tried:

import pandas as pd
from collections import defaultdict
d_new = defaultdict(pd.DataFrame)
for k_n in [1,3]:
    r = 1 if k_n == 1 else 3
    for k in range(r,r+2):
        d_new[k_n].append(d[k])

But this just throws up a dictionary of empty dataframes. We can convert the dataframes into lists and append them and create a dataframe from them later but I want to see if I can save that unnecessary step.

Answer 1

You can use itertools.groupby to group the keys (here I did it 2 by 2 from the key linear range) and pandas.concat to concatenate the dataframes:

from itertools import groupby
d_new = {2*k+1: pd.concat([d[e] for e in g])
         for k,g in groupby(d, lambda k: (k-1)//2)}

Output:

{1:    a  b
 0  1  3
 1  2  4
 0  3  5
 1  4  6,
 3:     a   b
 0  10  12
 1  11  13
 0  12  14
 1  13  15}

NB. I showed you the minimal main concept, but of course you can tweak it to your exact needs (eg, resetting the index, different condition for grouping)

If you have non linear range keys (ie not 1/2/3...), you'll need to wrap the dictionary in enumerate :

{2*k+1: pd.concat([x[1] for x in g])
 for k,g in groupby(enumerate(d.values()), lambda k: k[0]//2)}