Calculation between pandas dataframe returns NaN

Question

I have a pandas dataframe called df_mod . One variable of interest in this dataframe is called Evap_mod . When I use the command print(df_mod['Evap_mod']) , it returns:

2003-12-20 00:30:00    1.930664
2003-12-21 00:30:00    1.789290
2003-12-22 00:30:00    2.318347
2003-12-23 00:30:00    1.741943
2003-12-24 00:30:00    1.686124
2003-12-25 00:30:00    1.852876
2003-12-26 00:30:00    1.759650
2003-12-27 00:30:00    1.566521
2003-12-28 00:30:00    1.496039
2003-12-29 00:30:00    1.540751
2003-12-30 00:30:00    2.006475
2003-12-31 00:30:00    1.920912
Name: Evap_mod, Length: 729, dtype: float32

I have another pandas dataframe called dff . One variable of interest in this dataframe is called PET_PT . When I use the command print(dff['PET_PT']) , it returns:

2003-12-20    4.810697
2003-12-21    4.739378
2003-12-22    4.994467
2003-12-23    5.138086
2003-12-24    5.024226
2003-12-25    4.937206
2003-12-26    4.551416
2003-12-27         NaN
2003-12-28         NaN
2003-12-29         NaN
2003-12-30         NaN
2003-12-31         NaN
Freq: D, Name: PET_PT, Length: 729, dtype: float64

I would like to run the simple following calculation between those 2 variables:

df_mod['ER_mod']=(df_mod['Evap_mod']+np.mean(ddf['PET_PT']))/(ddf['PET_PT']+np.mean(ddf['PET_PT']))

Unfortunately, this calculation just returns NaN:

2003-12-20 00:30:00   NaN
2003-12-21 00:30:00   NaN
2003-12-22 00:30:00   NaN
2003-12-23 00:30:00   NaN
2003-12-24 00:30:00   NaN
2003-12-25 00:30:00   NaN
2003-12-26 00:30:00   NaN
2003-12-27 00:30:00   NaN
2003-12-28 00:30:00   NaN
2003-12-29 00:30:00   NaN
2003-12-30 00:30:00   NaN
2003-12-31 00:30:00   NaN
Name: ER_mod, Length: 729, dtype: float64

Does anyone has an idea why it returns NaN and how to overcome this issue?

Answer 1

Reason is different index values, so after divide index values not matched and created NaN s.

Solution is map Series ddf['PET_PT'] by helper column date created by DatetimeIndex.normalize for remove times and also use pandas mean s functions:

#same index values like df_mod
new = df_mod.assign(date = df_mod.index.normalize())['date'].map(ddf['PET_PT'])
print (new)
2003-12-20 00:30:00    4.810697
2003-12-21 00:30:00    4.739378
2003-12-22 00:30:00    4.994467
2003-12-23 00:30:00    5.138086
2003-12-24 00:30:00    5.024226
2003-12-25 00:30:00    4.937206
2003-12-26 00:30:00    4.551416
2003-12-27 00:30:00         NaN
2003-12-28 00:30:00         NaN
2003-12-29 00:30:00         NaN
2003-12-30 00:30:00         NaN
2003-12-31 00:30:00         NaN
Name: date, dtype: float64

df_mod['ER_mod']= df_mod['Evap_mod'] + ddf['PET_PT'].mean())/(new+ddf['PET_PT'].mean()
print (df_mod)
                     Evap_mod    ER_mod
2003-12-20 00:30:00  1.930664  0.702960
2003-12-21 00:30:00  1.789290  0.693480
2003-12-22 00:30:00  2.318347  0.729125
2003-12-23 00:30:00  1.741943  0.661170
2003-12-24 00:30:00  1.686124  0.663134
2003-12-25 00:30:00  1.852876  0.685986
2003-12-26 00:30:00  1.759650  0.704152
2003-12-27 00:30:00  1.566521       NaN
2003-12-28 00:30:00  1.496039       NaN
2003-12-29 00:30:00  1.540751       NaN
2003-12-30 00:30:00  2.006475       NaN
2003-12-31 00:30:00  1.920912       NaN

If same length DataFrame and only difference in inde values are times, you can reassign one index to another:

ddf.index = df_mod.index

df_mod['ER_mod'] = (df_mod['Evap_mod'] + ddf['PET_PT'].mean())/\
                   (ddf['PET_PT'] + ddf['PET_PT'].mean())
print (df_mod)
                     Evap_mod    ER_mod
2003-12-20 00:30:00  1.930664  0.702960
2003-12-21 00:30:00  1.789290  0.693480
2003-12-22 00:30:00  2.318347  0.729125
2003-12-23 00:30:00  1.741943  0.661170
2003-12-24 00:30:00  1.686124  0.663134
2003-12-25 00:30:00  1.852876  0.685986
2003-12-26 00:30:00  1.759650  0.704152
2003-12-27 00:30:00  1.566521       NaN
2003-12-28 00:30:00  1.496039       NaN
2003-12-29 00:30:00  1.540751       NaN
2003-12-30 00:30:00  2.006475       NaN
2003-12-31 00:30:00  1.920912       NaN

Answer 2

您的列包含缺失的数据，因此您应该根据您的目标通过不同的方法（平均值，零，中位数，随机等）来估算值

Answer 3

Here there is a difference between pandas and numpy behaviour. Whenever you compute np.mean(x) if x contains NaN you are going to have NaN as a result while working with pandas NaN are ignored. The following should work

df_mod['ER_mod'] = (df_mod['Evap_mod'] + ddf['PET_PT'].mean())/\
                   (ddf['PET_PT'] + ddf['PET_PT'].mean())

Otherwise you can use np.nanmean instead of np.mean .