在列标签上合并 Pandas 数据框并覆盖匹配行中的其他值

Question

I have these dataframes:

rec = pd.DataFrame({'batch': ["001","002","003"], 
                    'A': [1, 2, 3], 
                    'B': [4, 5, 6]})

ing1 = pd.DataFrame({'batch': ["002","003","004"], 
                     'C': [12, 13, 14], 
                     'D': [15, 16, 17], 
                     'E': [18, 19, 10]})

ing2 = pd.DataFrame({'batch': ["001","011","012"],
                     'C': [20, 21, 22], 
                     'D': [23, 24, 25], 
                     'F': [26, 27, 28]})

What I want is the following merged dataset, where columns with the same label are overwritten by the later merged dateset, and new columns are created for non-existing labels.

  batch  A  B   C   D     E     F
0   001  1  4  20  23   NaN  26.0
1   002  2  5  12  15  18.0   NaN
2   003  3  6  13  16  19.0   NaN

I have tried to merge rec with ing1 first:

final = pd.merge(rec, ing1, how ='left', on='batch', sort=False)

Intermediate result:

  batch  A  B     C     D     E
0   001  1  4   NaN   NaN   NaN
1   002  2  5  12.0  15.0  18.0
2   003  3  6  13.0  16.0  19.0

Then I merge a second time with ing2 , to obtain the missing information in columns C, D and E.

final = pd.merge(final, ing2, how ='left', on='batch', sort=False)

Result (not as expected):

  batch  A  B   C_x   D_x     E   C_y   D_y     F
0   001  1  4   NaN   NaN   NaN  20.0  23.0  26.0
1   002  2  5  12.0  15.0  18.0   NaN   NaN   NaN
2   003  3  6  13.0  16.0  19.0   NaN   NaN   NaN

I have also tried merge , concat , and combinefirst , however these seem to operate where they append the data from the second table onto the primary table. The only approach I can think of is to split the dataframe into rows that need to pull data from ing1 and rows that need ing2 , then append them to each other for the final dataset.

Answer 1

How about just applying np.where() after merging? If the right column (with suffix "_y") is not NA then take the right, else take the left.

final = rec.merge(ing1, how='left', on='batch')\
           .merge(ing2, how='left', on='batch')
final[["C", "D"]] = np.where(~final[["C_y", "D_y"]].isna(), final[["C_y", "D_y"]], final[["C_x", "D_x"]])

Output

print(final[["A","B","C","D","E","F"]])

   A  B     C     D     E     F
0  1  4  20.0  23.0   NaN  26.0
1  2  5  12.0  15.0  18.0   NaN
2  3  6  13.0  16.0  19.0   NaN

Answer 2

Actually, df.update() may be the conceptually closest function to what you're asking for. However, you have to set index and pre-allocate the output dataframe in advance. This may or may not cause more trouble than .merge() .

Code:

# set index
rec.set_index("batch", inplace=True)
ing1.set_index("batch", inplace=True)
ing2.set_index("batch", inplace=True)

# preallocate
final = pd.DataFrame(columns=["A","B","C","D","E","F"], index=rec.index)
# update in order
final.update(rec)
final.update(ing1)
final.update(ing2)

Result:

print(final)

       A  B   C   D    E    F
batch                        
001    1  4  20  23  NaN   26
002    2  5  12  15   18  NaN
003    3  6  13  16   19  NaN