通过min-max kwargs过滤熊猫数据框

Question

I have a function that has optional kwargs (8 total) based off min and max values entered by the user.

For example GR_min, GR_max, GR_N_min, GR_N_max, Hi_min, Hi_max etc...where the dataframe columns are GR , GR_N , Hi etc...

I'd like the dataframe to filter on the given min and max values but if one or more of the values are not given in the call of the function to set the default min-max values to just the min-max of the columns.

For example some pseudo code:

df = pd.DataFrame({'GR': [1, 2, 3, 4, 2, 3], 
'GR_N': [0.8, 0, 1, 0.6, 0.9, 1], 'Hi':[3, 6, 2, 5, 22, 7]})

Gets me:

    GR  GR_N    Hi
0   1   0.8     3
1   2   0.0     6
2   3   1.0     2
3   4   0.6     5
4   2   0.9     22
5   3   1.0     7

I want a function that does something like this:

def picker(data, **kwargs):

      data_filtered = data[data['GR'].between(GR_min, GR_max) &
                         data['GR_N'].between(GR_N_min, GR_N_max) &
                         data['Hi'].between(Hi_min, Hi_max)]

      return data_filtered

With an output after calling to be:

picker(data=df, GR_min=2, GR_max=3, Hi_min=1, Hi_max=6)

    GR  GR_N    Hi
1   2   0.0     6
2   3   1.0     2

Except instead of explicitly calling each column of the dataframe we use the **kwargs themselves to filter on.

Is there any way to do this?

Answer 1

DataFrame.query can be handy here, because it will parse a string containing conditions. So it will be enough to build a condition string from the keyword parameters.

Each individual condition could be built as: K<=val for a K_max=val parameter, and K>=val for a K_min=val parameter. To build the list, each individual condition must be enclosed in parentheses ( () ) and then joined with & .

Code could be:

def picker(data, **kwargs):
    def make_cond(k,v):
        if len(k)<5:
            raise(ValueError('Arg too short {}'.format(k)))
        if k.endswith('_min'):
            return '({}>={})'.format(k[:-4], v)
        elif k.endswith('_max'):
            return '({}<={})'.format(k[:-4], v)
        else:
            raise(ValueError('Unknow arg {}'.format(k)))
    strcond='&'.join((make_cond(k, v) for k,v in kwargs.items()))
    # print(strcond)     # uncomment for traces
    return data.query(strcond)

Answer 2

You could have a default dictionary for your kwargs specifying the min and max as -infinity and +infinity, and then just over-ride these with the user input. Something like this:

import numpy as np
def picker(data, **kwargs):
    d = dict(GR_min=-np.inf, GR_max=np.inf) # ... etc
    kwargs = {**d, **kwargs}
    data_filtered = data[data['GR'].between(kwargs["GR_min"], kwargs["GR_max"])] # ... etc
    return data_filtered

Answer 3

I'm a bit perplexed by this, filtering based on the min-max values in the columns would just be not filtering at all, no? Why not just only filter based on the arguments provided? Regardless, this sounds like a case for default arguments.

#create the DataFrame
df = pd.DataFrame({'GR': [1, 2, 3, 4, 2, 3], 
'GR_N': [0.8, 0, 1, 0.6, 0.9, 1], 'Hi':[3, 6, 2, 5, 22, 7]})

def picker(df, GR_min = None, GR_max = None, GR_N_min = None, GR_N_max = None,
           Hi_min = None, Hi_max = None): #use default arguments

           if GR_min == None:
               GR_min = df['GR'].min()
           if GR_max == None:
               GR_max = df['GR'].max()
           if GR_N_min == None:
               GR_N_min = df['GR_N'].min()
           if GR_N_max == None:
               GR_N_max == df['GR_N'].max()

           #filter the DataFrame with masks
           df_out = df.loc[(df['GR'] > GR_min) & (df['GR'] < GR_max) & 
                           (df['GR_N'] > GR_N_min) & (df['GR_N'] < GR_N_max)]
           return df_out