在python pandas的循环中合并大数据帧中的许多子数据帧

Question

My program will generate many small data frames such as the ones below

       Column_A  Column_B
 row1         1         2

       Column_A  Column_B
 row2         3         4

       Column_C  Column_D
 row1         5         6

       Column_C  Column_D
 row2         7         8

I want them to be merged as

       Column_A  Column_B  Column_C  Column_D
 row1         1         2         5         6
 row2         3         4         7         8

How can this be done when the dataframes need to be merged one at a time. The code to generate the smaller data frames are

 df = {}
 df[0] = pd.DataFrame({'Column_A' : [1], 
                       'Column_B' : [2]},
                      index = ["row1"])
 df[1] = pd.DataFrame({'Column_A' : [3],
                       'Column_B' : [4]},
                       index = ["row2"])
 df[2] = pd.DataFrame({'Column_C' : [5],
                       'Column_D' : [6]},
                      index = ["row1"]) 
 df[3] = pd.DataFrame({'Column_C' : [7],
                       'Column_D' : [8]},
                      index = ["row2"])

I tried using merge and concat but they always end up creating more columns by appending the existing columns with either _x, _y or just repeating the columns

For example, the merge in following way

pdf = pd.DataFrame()

for i in range(4):
    pdf = pdf.merge(pd.DataFrame(df[i], index=["row{}".format(((i)%2)+1)]), how='outer', left_index=True, right_index=True)

produces

      Column_A_x  Column_B_x  Column_A_y  Column_B_y  Column_C_x  Column_D_x  \
row1         1.0         2.0         NaN         NaN         5.0         6.0   
row2         NaN         NaN         3.0         4.0         NaN         NaN   

      Column_C_y  Column_D_y  
row1         NaN         NaN  
row2         7.0         8.0  

Can someone help me with the correct way to merge it

Answer 1

It would help you a lot, if you can in any way keep the left and right parts in separate containers. Eg columns A and B in one, columns C and D in the other. That way you could piece them together using pandas.concat quite easily. After the 2 halves have been built you need to merge them using the index in this case.

With your original df dictionary:

In [11]: pd.concat([df[0], df[1]]).merge(pd.concat([df[2], df[3]]), left_index=True, right_index=True)
Out[11]: 
      Column_A  Column_B  Column_C  Column_D
row1         1         2         5         6
row2         3         4         7         8

With containers for left and right halves the code reads a bit better (and there's no need for a loop):

left = [pd.DataFrame({'Column_A' : [1], 
                      'Column_B' : [2]},
                     index = ["row1"]),
        pd.DataFrame({'Column_A' : [3],
                      'Column_B' : [4]},
                     index = ["row2"])]

right = [pd.DataFrame({'Column_C' : [5],
                       'Column_D' : [6]},
                      index = ["row1"]),
         pd.DataFrame({'Column_C' : [7],
                       'Column_D' : [8]},
                      index = ["row2"])]

df = pd.concat(left).merge(pd.concat(right), left_index=True, right_index=True)

Finally if you truly have no option but to store them in a dictionary like in your example:

from functools import reduce, partial
from itertools import groupby

pdf = reduce(
    partial(pd.merge, left_index=True, right_index=True, how='outer'),
    (pd.concat(list(g))
     for cols, g in groupby(sorted(df.values(),
                                   key=lambda df_: tuple(df_.columns)),
                            lambda df_: tuple(df_.columns)))
)

Answer 2

try this:

In [186]: result = pd.concat([df[key].reset_index() for key in df.keys()],
   .....:                    ignore_index=True) \
   .....:            .set_index('index') \
   .....:            .groupby(level=0) \
   .....:            .sum() \
   .....:            .astype(int)

In [187]: result
Out[187]:
       Column_A  Column_B  Column_C  Column_D
index
row1          1         2         5         6
row2          3         4         7         8