将数据框分组并按组计算归一化标准偏差？

Question

I've got a dataframe that looks like this:

           product  cost_per_quantity
12779  0101010G0BB         160.788357
12653  0101010G0BC         179.493956
10390  0101010I0AA           0.425916
20361  0101010I0AA           0.603650
22504  0101010I0AA           0.633082

created with:

df = pd.DataFrame({ 'product': ['0101010G0BB', '0101010G0BC', '0101010I0AA', '0101010I0AA', '0101010I0AA'], 'cost_per_quantity': [160.788357, 179.493956, 0.425916, 0.603650, 0.633082]})

Now I want to find the products with the maximum variation in cost_per_quantity .

So for example, I'd like to examine the product 0101010I0AA and find the normalised standard deviation for cost_per_quantity across its three entries, and then compare it with normalised standard deviation for other products.

What's the best way to approach this? I tried:

df1 = df.groupby('product').agg(np.std)

but that just gives me a bunch of NaN s.

Answer 1

For aggregation df.groupby('product').agg(np.std) is correct but for 1-observation groups this returns NaN as the sample standard deviation cannot be calculated for 1-observation groups. Numpy default for standard deviation is population standard deviation but I guess Pandas is overriding that.

You can go with the population standard deviation to get 0 for those groups.

If you want to see the relative deviation with respect to the mean, you can use coefficient of variation :

df.groupby('product').apply(lambda x: np.std(x) / np.mean(x))

Now that np.std is in a lambda function, it behaves as expected.