append row values from one df to another if no duplicates in pandas

Question

I have theses two dfs

df1 = pd.DataFrame({'pupil': ["sarah", "john", "fred"],
                  'class': ["1a", "1a", "1a"]})


df2 = pd.DataFrame({'pupil_mixed': ["sarah", "john", "lex"],
                  'class': ["1a", "1c", "1a"]})

I want to append the row values from the column "pupil_mixed" from df2 to the column "pupil" in df1 if the values are no duplicates

desired outcome:

df1 = pd.DataFrame({'pupil': ["sarah", "john", "fred", 'lex'],
                  'class': ["1a", "1a", "1a", NaN]})

I used append with loc

df1 = df1.append(df2.loc[df2['pupil_mixed'] != df1['pupil'] ])

which just appended the other column to the df with the matching row value and changed the non matching row values to NaN

    pupil   class   pupil_mixed
0   sarah   1a      NaN
1   john    1a      NaN
2   fred    1a      NaN
2   NaN     1a      lex

Answer 1

You could use concat + drop_duplicates :

res = pd.concat((df1, df2['pupil_mixed'].to_frame('pupil'))).drop_duplicates('pupil')

print(res)

Output

   pupil class
0  sarah    1a
1   john    1a
2   fred    1a
2    lex   NaN

As an alternative you could filter first (with isin ) and then concat:

# filter the rows in df2, rename the column pupil_mixed
filtered = df2.loc[~df2['pupil_mixed'].isin(df1['pupil'])]

# create a new single column DataFrame with the pupil column
res = pd.concat((df1, filtered['pupil_mixed'].to_frame('pupil')))

print(res)

Both solutions use to_frame , with the name parameter, effectively changing the column name.

Answer 2

# distinct df1 & df2
df1['tag'] = 1
df2['tag'] = 2

# change the column name the same
df2.columns = df1.columns
df1 = df1.append(df2)
# drop_duplicates by keep df1
df1 = df1.drop_duplicates('pupil', keep='first')

# set tag == 2, class is null
cond = df1['tag'] == 2
df1.loc[cond, 'class'] = np.nan
del df1['tag']

print(df1)

output:

print(df1)

   pupil class
0  sarah    1a
1   john    1a
2   fred    1a
3    lex   NaN

Answer 3

You could use a merge, after renaming pupil_mixed in df2:

df1.merge(df2["pupil_mixed"].rename("pupil"), how="outer")

   pupil    class
0   sarah   1a
1   john    1a
2   fred    1a
3   lex    NaN