pandas - partially updating DataFrame with derived calculations of a subset groupby

Question

I have a DataFrame with some NaN records that I want to fill based on a combination of data of the NaN record (index in this example) and of the non-NaN records. The original DataFrame should be modified.

Details of input/output/code below:

I have an initial DataFrame that contains some pre-calculated data:

Initial Input

raw_data = {'raw':[x for x in range(5)]+[np.nan for x in range(2)]}
source = pd.DataFrame(raw_data)

  raw
0 0.0
1 1.0
2 2.0
3 3.0
4 4.0
5 NaN
6 NaN

I want to identify and perform calculations to "update" the NaN data, where the calculations are based on data of the non-NaN data and some data from the NaN records.

In this contrived example I am calculating this as:

• Calculate average/mean of 'valid' records.
• Add this to the index number of 'invalid' records.

Finally this needs to be updated on the initial DataFrame.

Desired Output

  raw  valid
0 0.0      1
1 1.0      1
2 2.0      1
3 3.0      1
4 4.0      1
5 7.0      0
6 8.0      0

The current solution I have (below) makes a calculation on a copy then updates the original DataFrame.

# Setup grouping by NaN in 'raw'
source['valid'] = ~np.isnan(source['raw'])*1
subsets = source.groupby('valid')

# Mean of 'valid' is used later to fill 'invalid' records
valid_mean = subsets.get_group(1)['raw'].mean()

# Operate on a copy of group(0), then update the original DataFrame
invalid = subsets.get_group(0).copy()
invalid['raw'] = subsets.get_group(0).index + valid_mean
source.update(invalid)

Is there a less clunky or more efficient way to do this? The real application is on significantly larger DataFrames (and with a significantly longer process of processing NaN rows).

Thanks in advance.

Answer 1

You can use combine_first :

#mean by default omit `NaN`s
m = source['raw'].mean()
#same as
#m = source['raw'].dropna().mean()
print (m)
2.0

#create valid column if necessary
source['valid'] = source['raw'].notnull().astype(int)
#update NaNs
source['raw'] = source['raw'].combine_first(source.index.to_series() + m)

print (source)
   raw  valid
0  0.0      1
1  1.0      1
2  2.0      1
3  3.0      1
4  4.0      1
5  7.0      0
6  8.0      0