Filling empty values in one dataframe based on column in another dataframe?

Question

I have a first dataframe, which includes some missing values in a column. I then have a second dataframe, which includes a more complete dataset, but not necessarily at the same exact indices. As an example, here's a depiction of the situation:

It's clear that filling for the indices that match is easy (eg, the first nan can be filled with 634 from the second dataframe). For the indices that aren't in the other DF, I would like to interpolate between the two nearest values (eg, to fill the 5.0, I would like to interpolate between 4.8 and 5.2 in df2). I'm not sure how to do this, at least not in a pandas way. My instinct is to iterate through the missing values, manually find the closest index in df2 and then interpolate between. I'm sure there's a smarter way of going about this though. Any tips?

Answer 1

I changed column name Index -> arg to avoid confusion.

First load data frames

df1 = pd.DataFrame({
    'arg': {0: 1.0, 1: 2.3, 2: 2.5, 3: 3.6, 4: 5.0, 5: 5.9, 6: 6.0, 7: 6.2, 8: 6.3, 9: 6.4},
    'value': {0: 634.0, 1: 500.0, 2: 439.0, 3: 287.0, 4: 641.0, 5: 212.0, 6: 374.0, 7: 358.0, 8: 600.0, 9: 755.0}
}) 
df2 = pd.DataFrame({
    'arg': {0: 1.0, 1: 1.4, 2: 1.8, 3: 2.2, 4: 2.4, 5: 2.8, 6: 3.2, 7: 3.6, 8: 4.0, 9: 4.4, 10: 4.8, 11: 5.2, 12: 5.6, 13: 6.0, 14: 6.4},
    'value': {0: 634, 1: 8, 2: 218, 3: 813, 4: 338, 5: 339, 6: 935, 7: 287, 8: 376, 9: 481, 10: 727, 11: 555, 12: 50, 13: 374, 14: 755}
})

Calculate left join on df1 and update values from df1 to df2.

temp = df1.merge(df2, on="arg", how="left")
df1["value"] = temp.value_y.combine_first(temp.value_x)

get still NaN values

to_interpolate = df1[df1.value.isna()]

add arguments without values to df2 and interpolate their values.

df3 = pd.concat([to_interpolate, df2]).sort_values("arg")
df3.value.interpolate(inplace=True)

repeat merging.

temp = df1.merge(df3, on="arg", how="left")
df1["value"] = temp.value_x.combine_first(temp.value_y)
print(df1)

Outputs:

   arg  value
0  1.0  634.0
1  2.3  500.0
2  2.5  439.0
3  3.6  287.0
4  5.0  641.0
5  5.9  212.0
6  6.0  374.0
7  6.2  358.0
8  6.3  600.0
9  6.4  755.0