如果我不知道存在哪些列，如何使用 contains 对列列表进行过滤？

Question

I want to filter my dataframe to look for columns containing a known string. I know you can do something like this:

summ_proc = summ_proc[
                summ_proc['data.process.name'].str.contains(indicator) |         
                summ_proc['data.win.eventdata.processName'].str.contains(indicator) |
                summ_proc['data.win.eventdata.logonProcessName'].str.contains(indicator) | 
                summ_proc['syscheck.audit.process.name'].str.contains(indicator)
            ]

where I'm using the | operator to check against multiple columns. But there are cases where a certain column name isn't present. So 'data.process.name' might not be present every time.

I tried the following implementation:

summ_proc[summ_proc.apply(lambda x: summ_proc['data.process.name'].str.contains(indicator) if 'data.process.name' in summ_proc.columns else summ_proc)]

And that works. But I'm not sure how I can apply the OR operator to this lambda function. I want all the rows where either data.process.name or data.win.eventdata.processName or data.win.eventdata.logonProcessName or syscheck.audit.process.name contains the indicator.

EDIT:

I tried the following approach, where I created individual frames and concated all the frames.

summ_proc1 = summ_proc[summ_proc.apply(lambda x: summ_proc['data.process.name'].str.contains(indicator) if 'data.process.name' in summ_proc.columns else summ_proc)]
summ_proc2 = summ_proc[summ_proc.apply(lambda x: summ_proc['data.win.eventdata.processName'].str.contains(indicator) if 'data.win.eventdata.processName' in summ_proc.columns else summ_proc)]
summ_proc3 = summ_proc[summ_proc.apply(lambda x: summ_proc['data.win.eventdata.logonProcessName'].str.contains(indicator) if 'data.win.eventdata.logonProcessName' in summ_proc.columns else summ_proc)]
frames = [summ_proc1, summ_proc2, summ_proc3]
result = pd.concat(frames)

This works, but I'm curious if there's a better more pythonic approach? Or if this current method will cause more downstream issues?

Answer 1

should work with something like this:

import numpy as np


columns = ['data.process.name', 'data.win.eventdata.processName']

# filter columns that are in summ_proc
available_columns = [c for c in columns if c in summ_proc.columns]

# array of Boolean values indicating if c contains indicator
ss = [summ_proc[c].str.contains(indicator) for c in available_columns]

# reduce without '|' by using 'np.logical_or'
indexer = np.logical_or.reduce(ss)

result = summ_proc[indexer]