Pandas - 在不同数据帧的两列之间“选择条件存在的位置”

Question

I have two dataframes:

data1 = {
    'id': [1,1,2,2],
    'tag': [700,800,700,800],
    'Membership': [1,0.9,0.8,0.7],
}
data2 = {
    'id': [1,2,3,3],
    'tag': [700,800,600,500],
    'Membership': [0.5,0.9,0.8,0.7],
}
df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

Which looks like:

>>> df1
      id   tag  Membership
0      1   700         1.0
1      1   800         0.9
2      2   700         0.8
3      2   800         0.7

>>> df2
      id   tag  Membership
0      1   700         0.5
1      2   800         0.9
2      3   600         0.8
3      3   500         0.7

I want to add rows from df1 to df2, where the combination of (id,tag) doesn't exist in df2 . So, any rows without the condition of df1['id'] == df2['id'] and df1['tag'] == df2['tag'] should be added to df2:

>>> df2
      id   tag  Membership
0      1   700         0.5
1      2   800         0.9
2      3   600         0.8
3      3   500         0.7
4      1   800         0.9  # This row added
5      2   700         0.8  # This row added

What I've tried:

I tried to find the rows where my condition is not true, then appending the result to df2:

new_rows = df1[~((df1['id'] == df2['id']) & (df1['tag'] == df2['tag']))]
df2 = df2.append(new_rows).reset_index(drop=True)

But as you can see, I'm getting wrong result, because (id,tag) pair of (2,800) already exists in df2. why is that?

>>> df2
      id   tag  Membership
0      1   700         0.5
1      2   800         0.9
2      3   600         0.8
3      3   500         0.7
4      1   800         0.9  # correct
5      2   700         0.8  # correct
6      2   800         0.7  # THIS SHOULDN't BE ADDED

Answer 1

Solution using combine_first :

indices = ['id', 'tag']
left = df2.set_index(indices)
right = df1.set_index(indices)
combined = left.combine_first(right).reset_index()
combined

Result:

   id  tag  Membership
0   1  700         0.5
1   1  800         0.9
2   2  700         0.8
3   2  800         0.9
4   3  500         0.7
5   3  600         0.8

Answer 2

The equality operator that you're using in this condition:

(df1['id'] == df2['id']) & (df1['tag'] == df2['tag'])

is not the right tool for this job, it doesn't work like you expect, it compares the dataframes row by row. Let's start with a simpler case:

In [5]: df1['id'] == df2['id']
Out[5]: 
0     True
1    False
2    False
3    False
Name: id, dtype: bool

Id 1 is found on row 0 in both series, so you get True. Id 2 is present in both series, but never on the same row, the position never matches. Same for the tag :

In [6]: df1['tag'] == df2['tag']
Out[6]: 
0     True
1     True
2    False
3    False
Name: tag, dtype: bool

So when you combine with & , only the first row matches:

In [7]: (df1['id'] == df2['id']) & (df1['tag'] == df2['tag'])
Out[7]: 
0     True
1    False
2    False
3    False
dtype: bool

which is why the (id,tag) pair of (2,800) is not recognized as being already present.

So instead of the equality operator, you should use merge as suggested by the other answers.

Answer 3

Here you go:

df3 = pd.merge(df2,df1,on=['id','tag'],how='outer')
df3 = df3[df3.Membership_x.isna()][['id','tag','Membership_y']].rename(columns={'Membership_y':'Membership'})
df3 = df2.append(df3)
df3

Prints

id  tag Membership
0   1   700 0.5
1   2   800 0.9
2   3   600 0.8
3   3   500 0.7
4   1   800 0.9
5   2   700 0.8