从符合特定标准的观察中创建新的 Pandas Dataframe

Question

I have two original dataframes. One contains limits: df_limits

           feat_1   feat_2  feat_3  
target       12       9       90    
UL           15       10      120   
LL           9        8       60

where target is ideal value, UL - upper limit, LL - lower limit

And another one original data: df_to_check

ID          feat_1  feat_2  feat_3  
123         12.5    9.6     100 
456         18      3       100
789         9       11      100

I'm creating a function which desired output is get ID and features which are below or above the threshold (limits from first Dataframe) Till now I'm able to recognise which features are out of limits but I'm getting full output of original Dataframe...

def table(df_limits, df_to_check, column):
    
    UL = df_limits[column].loc['target'] + df_limits[column].loc['UL']
    LL = df_limits[column].loc['target'] + df_limits[column].loc['LL']
    
    UL_index = df_to_check.loc[df_to_check[column] > UL].index
    LL_index = df_to_check.loc[df_to_check[column] < LL].index
    
    if UL_index is not None:
        above_limit = {'ID': df_to_check['ID'],
                       'column': df_to_check[column],
                       'target':  df_limits[column].loc['target']}
        
    return pd.DataFrame(above_limit)

What I should change so my desired output would be: (showing only ID and column where observations are out of limit)

The best if it would show also how many percent of original value is deviate from ideal value target (I would be glad for advices how to add such a column)

ID     column    target    value    deviate(%)
456    feat_1    12        18       50
456    feat_2    9         3        ...
789    feat_2    9         11       ...

Now after running this function its returning whole dataset because statement says if not null... which is not null... I understand why I have this issue but I don't know how to change it

Issue is with statement if UL_index is not None: since it returning whole dataset and I'm looking for way how to replace this part

Answer 1

First of all, you have not provided a reproducible example https://stackoverflow.com/help/minimal-reproducible-example because you have not shared the code which produces the two initial dataframes. Next time you ask a question, please keep it in mind, Without those, I made a toy example with my own (random) data.

I start by unpivoting what you call dataframe_to_check: that's because, if you want to check each feature independently, then that dataframe is not normalised (you might want to look up what database normalisation means).

The next step is a left outer join between the unpivoted dataframe you want to check and the (transposed) dataframe with the limits.

Once you have that, you can easily calculate whether a row is within the range, the deviation between value and target, etc, and you can of course group this however you want.

My code is below. It should be easy enough to customise it to your case.

import pandas as pd
import numpy as np


df_limits = pd.DataFrame(index =['min val','max val','target'])

df_limits['a']=[2,4,3]
df_limits['b']=[3,5,4.5]


df =pd.DataFrame(columns = df_limits.columns, data =np.random.rand(100,2)*6 )

df_unpiv = pd.melt( df.reset_index().rename(columns ={'index':'id'}), id_vars='id', var_name ='feature', value_name = 'value' )

# I reset the index because I couldn't get a join on a column and index, but there is probably a better way to do it
df_joined = pd.merge( df_unpiv, df_limits.transpose().reset_index().rename(columns = {'index':'feature'}), how='left', on ='feature' )
df_joined['abs diff from target'] = abs( df_joined['value'] - df_joined['target'] )

df_joined['outside range'] =  (df_joined['value'] < df_joined['min val'] ) | (df_joined['value'] > df_joined['max val'])
    
df_outside_range = df_joined.query(" `outside range` == True "  )
df_inside_range = df_joined.query(" `outside range` == False "  )

Answer 2

Approach

• reshape
• merge
• calculate

new_df = (df_to_check.set_index("ID").unstack().reset_index()
 .rename(columns={"level_0":"column",0:"value"})
 .merge(df_limits.T, left_on="column", right_index=True)
 .assign(deviate=lambda dfa: (dfa.value-dfa.target)/dfa.target)
)

column	ID	value	target	UL	LL	deviate
feat_1	123	12.5	12	15	9	0.0416667
feat_1	456	18	12	15	9	0.5
feat_1	789	9	12	15	9	-0.25
feat_2	123	9.6	9	10	8	0.0666667
feat_2	456	3	9	10	8	-0.666667
feat_2	789	11	9	10	8	0.222222
feat_3	123	100	90	120	60	0.111111
feat_3	456	100	90	120	60	0.111111
feat_3	789	100	90	120	60	0.111111

Answer 3

I solved my issue maybe in bit clumsy way but it works as desired... If someone have better answer I will still appreciate:

Example how to get only observations above limits, to have both just concatenate observation from UL_index and LL_index

def table(df_limits,df_to_check,column):
    
    above_limit = []
    df_above_limit = pd.DataFrame()
    
    UL = df_limits[column].loc['target'] + df_limits[column].loc['UL']
    LL = df_limits[column].loc['target'] + df_limits[column].loc['LL']
    
    UL_index = df_to_check.loc[df_to_check[column] > UL].index
    LL_index = df_to_check.loc[df_to_check[column] < LL].index
    
    df_to_check_UL = df_to_check.loc[UL_index]
    df_to_check_LL = df_to_check.loc[LL_index]
    
    above_limit = {
                    'ID': df_to_check_UL['ID'],
                    'feature value': df_to_check[column],
                    'target':  df_limits[column].loc['target']
    }
        
    df_above_limit = pd.DataFrame(above_limit, index = df_to_check_UL.index)
        
    return df_above_limit