创建熊猫数据透视表时存储所有值

Question

Basically, I'm aggregating prices over three indices to determine: mean, std, as well as an upper/lower limit. So far so good. However, now I want to also find the lowest identified price which is still >= the computed lower limit.

My first idea was to use np.min to find the lowest price -> this obviously disregards the lower-limit and is not useful. Now I'm trying to store all the values the pivot table identified to find the price which still is >= lower-limit. Any ideas?

pivot = pd.pivot_table(temp, index=['A','B','C'],values=['price'], aggfunc=[np.mean,np.std],fill_value=0)

pivot['lower_limit'] = pivot['mean'] - 2 * pivot['std']
pivot['upper_limit'] = pivot['mean'] + 2 * pivot['std']

Answer 1

First, merge pivoted[lower_limit] back into temp . Thus, for each price in temp there is also a lower_limit value.

temp = pd.merge(temp, pivoted['lower_limit'].reset_index(), on=ABC)

Then you can restrict your attention to those rows in temp for which the price is >= lower_limit :

temp.loc[temp['price'] >= temp['lower_limit']]

The desired result can be found by computing a groupby/min :

result = temp.loc[temp['price'] >= temp['lower_limit']].groupby(ABC)['price'].min()

---

For example,

import numpy as np
import pandas as pd

np.random.seed(2017)
N = 1000
ABC = list('ABC')
temp = pd.DataFrame(np.random.randint(2, size=(N,3)), columns=ABC)
temp['price'] = np.random.random(N)
pivoted = pd.pivot_table(temp, index=['A','B','C'],values=['price'], 
                         aggfunc=[np.mean,np.std],fill_value=0)
pivoted['lower_limit'] = pivoted['mean'] - 2 * pivoted['std']
pivoted['upper_limit'] = pivoted['mean'] + 2 * pivoted['std']

temp = pd.merge(temp, pivoted['lower_limit'].reset_index(), on=ABC)
result = temp.loc[temp['price'] >= temp['lower_limit']].groupby(ABC)['price'].min()
print(result)

yields

A  B  C
0  0  0    0.003628
      1    0.000132
   1  0    0.005833
      1    0.000159
1  0  0    0.006203
      1    0.000536
   1  0    0.001745
      1    0.025713