用单独的Pan​​das系列中的值替换NaN值（在Pandas DataFrame中）的最佳方法是什么？

Question

I started with a Pandas DataFrame which has a column with many NaN values.

I split this Pandas DataFrame into two DataFrames: non-NaN and NaN.

I estimated a linear regression model to try to fill in the NaN values (as a function of the other columns).

So I now have a separate Pandas Series that has the estimated values. Its length is the same length as the NaN DataFrame.

I now want to put these estimated values back into the NaN DataFrame, so that I can then ultimately pd.concat() these two DataFrames into one DataFrame that I can then use for my analysis.

I cannot figure out a way to put these values back into the NaN DataFrame into the correct rows. Every time I tried, only some of the NaNs get filled (and probably in the wrong order). It seems to be something to do with the way they're indexed.

df_nan["Column"] = y_predicted

This is the way I've tried to do it, but it only fills in some of the rows, and incorrectly. Something to do with indices maybe?

Answer 1

I think a way of doing this could be the following: you keep your raw dataframe and use apply on the column you want to impute.

df['imputed_column'] = df.apply(lambda x: x.Column if(pd.notnull(x.Column)) else y_predicted[x.name],axis=1)

The following line will get the estimated value if it has a null value (with x.name being the index of the row). Otherwise, it will keep the same value.