将新数据帧添加到现有数据库，但仅在列名匹配时添加

Question

I have two dataframes that I am trying to combine but I'm not getting the result I want using pandas.concat.

I have a database of data that I want to add new data to but only if the column of name matches.

Let says df1 is:

A B C D
1 1 2 2
3 3 4 4
5 5 6 6

and df2 is:

A E D F
7 7 8 8
9 9 0 0

the result I would like to get is:

A B C D
1 1 2 2
3 3 4 4
5 5 6 6
7 - - 8
9 - - 0

The blank data doesn't have to be - it can be anything.

When I use:

results = pandas.concat([df1, df2], axis=0, join='outer')

it gives me a new dataframe with all of the columns A through F, instead of what I want. Any ideas for how I can accomplish this? Thanks!

Answer 1

You want to use the pd.DataFrame.align method and specify that you want to align with the left argument's indices and that you only care about columns.

d1, d2 = df1.align(df2, join='left', axis=1)

Then you can use pd.DataFrame.append or pd.concat

pd.concat([d1, d2], ignore_index=True)

   A    B    C  D
0  1  1.0  2.0  2
1  3  3.0  4.0  4
2  5  5.0  6.0  6
3  7  NaN  NaN  8
4  9  NaN  NaN  0

Or

d1.append(d2, ignore_index=True)

   A    B    C  D
0  1  1.0  2.0  2
1  3  3.0  4.0  4
2  5  5.0  6.0  6
3  7  NaN  NaN  8
4  9  NaN  NaN  0

---

My preferred way would be to skip the reassignment to names

pd.concat(df1.align(df2, 'left', 1), ignore_index=True)

   A    B    C  D
0  1  1.0  2.0  2
1  3  3.0  4.0  4
2  5  5.0  6.0  6
3  7  NaN  NaN  8
4  9  NaN  NaN  0

Answer 2

You can use find the intersection of columns on df2 and concat or append :

pd.concat(
    [df1, df2[df1.columns.intersection(df2.columns)]]
)

Or,

df1.append(df2[df1.columns.intersection(df2.columns)])

   A    B    C  D
0  1  1.0  2.0  2
1  3  3.0  4.0  4
2  5  5.0  6.0  6
0  7  NaN  NaN  8
1  9  NaN  NaN  0

Answer 3

You can also use reindex and concat:

pd.concat([df1,df2.reindex(columns=df1.columns)])
Out[81]: 
   A    B    C  D
0  1  1.0  2.0  2
1  3  3.0  4.0  4
2  5  5.0  6.0  6
0  7  NaN  NaN  8
1  9  NaN  NaN  0

Answer 4

Transpose first before merging.

df1.T.merge(df2.T, how="left", left_index=True, right_index=True).T

    A   B   C   D
0_x 1.0 1.0 2.0 2.0
1_x 3.0 3.0 4.0 4.0
2   5.0 5.0 6.0 6.0
0_y 7.0 NaN NaN 8.0
1_y 9.0 NaN NaN 0.0

df1.T           df2.T

    0   1   2      1 2
A   1   3   5   A  7 9
B   1   3   5   E  7 9
C   2   4   6   D  8 0
D   2   4   6   F  8 0

Now the result can be obtained with a merge with how="left" and we use the indices as the join key by passing left_index=True and right_index=True .

df1.T.merge(df2.T, how="left", left_index=True, right_index=True)

    0_x 1_x 2   0_y 1_y
A   1   3   5   7.0 9.0
B   1   3   5   NaN NaN
C   2   4   6   NaN NaN
D   2   4   6   8.0 0.0