将列和值匹配到行的功能-DataFrames的交集

Question

Lets say I have the following DataFrame:

>>>df = pd.DataFrame([[5,2,3,11],[5,3,3,8],[9,4,11,12],[5,14,15,16]],columns=["a","b","c","d"])
>>>df
   a   b   c   d
0  5   2   3  11
1  5   3   3   8
2  9   4  11  12
3  5  14  15  16

If I wanted to match all the rows which had the column values: 'a' equal to 5, 'b' < 10 and 'c' greater than or equal to 3, I could just do the following:

df[(df['a'] == 5) & (df['b'] < 10) & (df['c'] >= 3)]

and that would give me the result I was after:

   a  b  c   d
0  5  2  3  11
1  5  3  3   8

Entering that code to match the rows was laborious so I decided to make a function called row_matcher which would take 2 arguments: a Pandas DataFrame and a list of lists of length 3 - column of choice, operator and value.

def get_operator_fn(op):
    import operator
    return {
        '<' : operator.lt,
        '<=' : operator.le,
        '==' : operator.eq,
        '!=' : operator.ne,
        '>=' : operator.ge,
        '>' : operator.gt,
        '&' : operator.and_
        }[op]

def row_matcher(df,parameters):
    import pandas as pd
    """Parameter should be [column,operation,value]
    Example: ['trial',"==",1]
    """
    operations = [df[get_operator_fn(operation)(df[column],value)] for column,operation,value in parameters]
    return reduce(lambda left,right: pd.merge(left,right,how='inner'), operations)

>>>row_matcher(df,[["a","==",5],["b","<",10],["c",">=",3]])

Unfortunately with this code it throws up an error for the return reduce(...) line: TypeError: Could not compare <type 'str'> type with Series

I tried replacing the return reduce(...) line with: df[reduce(operator.and_,operations)]

This still results in an error: TypeError: unsupported operand type(s) for &: 'str' and 'str'

I would appreciate any assistance.

Answer 1

I think this would be a lot simpler using the query() method. Using this, your initial example can be written as:

df.query('a==5 & b<10 & c>=3')

Honestly, if you use the query() method, I don't think you'd gain much from your function, unless you're reading in lots of conditions from an external file. If still want to write the row_matcher function, just use string joins to combine your list of lists as a single string, obeying the query() syntax. Once you have a single string, pass it along to the query() method.

You may need to install the numexpr module to use the query() method. You can get around this by supplying the keyword argument engine='python' to query() method. This may be less efficient than using the numexpr module, so it might be worthwhile to install the module if performance becomes an issue.