Pandas find and interpolate missing value

Question

This question is pretty much a follow up from Pandas pivot or reshape dataframe with NaN

When decoding videos some frames go missing and that data needs to be interpolated

Current df

frame   pvol    vvol    area    label
0       NaN     109.8   120     v
2       NaN     160.4   140     v
0       23.1    NaN     110     p
1       24.3    NaN     110     p
2       25.6    NaN     112     p

Expected df

frame   pvol    vvol    p_area  v_area
0       23.1    109.8   110     110
1       24.3    135.1   110     111 # Interpolated for label v
2       25.6    160.4   112     120

I know I can do df.interpolate() once the current_df is reshaped for only p frames. The reshaping is the issue.

Note: label p >= label v meaning label p will always have all the frames but v can have missed frames

Answer 1

You can reshape, dropna as in the previous question, except that now you need to specify that you want to drop only empty columns, then interpolate:

out = (df.pivot(index='frame', columns='label')
         .dropna(axis=1, how='all')            # only drop empty columns
         .interpolate()                        # interpolate
      )

out.columns = [f'{y}_{x}' for x,y in out.columns]

Output:

       p_pvol  v_vvol  p_area  v_area
frame                                
0        23.1   109.8   110.0   120.0
1        24.3   135.1   110.0   130.0
2        25.6   160.4   112.0   140.0

Answer 2

Change the dropna remove the issue

s = df.set_index(['frame','label']).unstack().dropna(thresh=1,axis=1)
s.columns = s.columns.map('_'.join)
s = s.interpolate()
Out[279]: 
       pvol_p  vvol_v  area_p  area_v
frame                                
0        23.1   109.8   110.0   120.0
1        24.3   135.1   110.0   130.0
2        25.6   160.4   112.0   140.0