熊猫dataframe.resample std仅在某些列上？

Question

I am trying to calculate monthly standard deviations for my data.

My data is loaded from a database into a dataframe with the following columns:

measurement_time, level, mixing_ratio, concentration

To calculate the monthly standard deviations, I do the following:

    df_std = df_std.set_index('measurement_time')
    df_std = df_std.groupby(['level'], as_index=False)
    df_std = df_std.resample('M').std()

The result is:

>> df_std.head()
                    level  mixing_ratio  concentration
  measurement_time                                    
0 2016-01-31          0.0  3.435376e-11       0.000015
  2016-02-29          0.0  2.692636e-11       0.000012
  2016-03-31          0.0  6.709993e-11       0.000029
  2016-04-30          0.0  3.338249e-11       0.000014
  2016-05-31          0.0  3.916523e-11       0.000017

The problem is that it is calculating the standard deviation on the level , too, while I only want the calculation performed on mixing_ratio & concentration

The result should be the monthly standard deviations at each level. If I had 7 levels, I would expect my dataframe to have 84 rows (7 * 12 months).

How can I fix this?

Answer 1

When you resample you have to do something with the aggregated data.

Example of different functions for different columns:

df_resample = df_std.resample('M').agg({'level': np.mean, 'mixing_ratio': np.std, 'concentration':np.std})

Other option is to groupby time and level.

Answer 2

SamuelNLP's answer is perfect for the case of different calculations on the DataFrame, as is presented in his example!

But I had a very similar problem as pookie, a large DataFrame (length>120'000'000), where I needed the std of all but one column over every 5 minute period. And in this specific case, there is a simple but faster alternative:

data_std = data.resample('5min').std()
data_std.drop('temperature', axis=1)

I just calculated the std after the resampling, and drop the columns that I didn't need.

>> data.head()
                                 x      y        z  temperature
time                
2018-02-21 11:00:06.354      0.606  0.764   -0.499  21.163203
2018-02-21 11:00:06.364     -0.762  0.127   -0.499  21.163203
2018-02-21 11:00:06.374     -0.793  0.143   -0.482  21.163203
2018-02-21 11:00:06.384     -0.809  0.064   -0.418  21.163203
2018-02-21 11:00:06.394     -0.825  -0.016  -0.401  21.163203

>> data_std = data.resample('5min').std()
>> data_std.drop('temperature', axis=1)
>> data_std.head()
                               x           y           z
time            
2018-02-21 11:00:05     0.260700    0.192227    0.244653
2018-02-21 11:00:10     0.125168    0.164327    0.116562
2018-02-21 11:00:15     0.138330    0.154963    0.126264
2018-02-21 11:00:20     0.182339    0.204350    0.226019
2018-02-21 11:00:25     0.193661    0.107022    0.133125

Speed-up: I tested the two options, over different DataFrame lengths, each 50 times with a simple time.time() measurement.

# resample-drop method
tmp = data.resample('5s').std()
tmp.drop('temperature', axis=1, inplace=True)

# aggregate method
tmp = data.resample('5s').agg({'x': np.std, 'y':np.std, 'z':np.std})

The resample-drop method is 20%-25% faster than the aggregate method.

I assume this runtime difference is due to a more efficient implementation of the simple resample-drop part as compared to the aggregate. But I would be glad for more explanations.