How to join two column in dataframe by overwriting only NaN values in pandas?

Question

I am trying to join two columns by overwriting only NaN values in the second column. I have tried multiple things but nothing is working.

                    New.Market.Cap  New.Market.Cap2  Expected.Output
Date       Symbol
2017-01-01 BTC       4.467053e+09              NaN    4.467053e+09  
           ETH       7.148243e+08     6.059076e+08    6.059076e+08 
           XRP       3.633730e+08              NaN    3.633730e+08 

2017-01-02 BTC       4.575871e+09              NaN    4.575871e+09
           ETH       7.334621e+08     6.249679e+08    6.249679e+08
           XRP       3.633730e+08              NaN    3.633730e+08

I've tried multiple things, but could not make it work.

Answer 1

Use Series.combine_first or Series.fillna :

df['Expected.Output'] = df['New.Market.Cap2'].combine_first(df['New.Market.Cap'])

Or:

df['Expected.Output'] = df['New.Market.Cap2'].fillna(df['New.Market.Cap'])

If need also remove columns DataFrame.pop is your friend:

df['Expected.Output'] = df.pop('New.Market.Cap2').combine_first(df.pop('New.Market.Cap'))