Python Pandas DF Replace NaN with value from other DF by Index

Question

I have two Dataframes where the Index can be set to ['Date', 'Name']. I now want to replace the NaN in the first Dataframe for all common columns with the Data which can be found in the second one (which can also have NaN values in the columns). They look like this:

NaN = np.nan
df1 = pd.DataFrame([
    ['2020-01-01', 'Foo1', 8, 0.999],
    ['2020-01-01', 'Bar1', NaN, NaN],
    ['2020-01-02', 'Foo1', 1, 0.564],
    ['2020-01-03', 'Foo1', NaN, NaN]],
    columns=['Date', 'Name', 'Val1', 'Val2'])

df2 = pd.DataFrame([
    ['2020-01-01', 'Foo1', 8, 0.999],
    ['2020-01-01', 'Bar1', 5, 0.6],
    ['2020-01-02', 'Foo1', 1, 0.564],
    ['2020-01-03', 'Foo1', NaN, NaN]],
    columns=['Date', 'Name', 'Val1', 'Val2'])

I tried to do it with where and replace statement but unfortunately I can't figure it out nor find anything which helped here so far. Thanks a lot in advance!

Answer 1

Try with combine_first or fillna

df1 = df1.set_index(['Date','Name']).combine_first(df2.set_index(['Date','Name'])).reset_index()
df1
         Date  Name  Val1   Val2
0  2020-01-01  Foo1   8.0  0.999
1  2020-01-01  Bar1   5.0  0.600
2  2020-01-02  Foo1   1.0  0.564
3  2020-01-03  Foo1   NaN    NaN

df1 = df1.set_index(['Date','Name']).fillna(df2.set_index(['Date','Name'])).reset_index()