根据不同列子集的组合删除具有 NaN 的行

Question

I would like to drop all rows with NaN values based on combinations of columns subsets. Let's demonstrate this on simple example:

df = pd.DataFrame(
    [[1, np.nan, 3, 4], [1, 2, 3, 4], [np.nan, np.nan, 3, 4], [1, np.nan, np.nan, 4], [1, 2, np.nan, np.nan]],
    columns=["a1", "a2", "b1", "b2"],
)

print(df)
#    a1   a2   b1   b2
# 0  1.0  NaN  3.0  4.0
# 1  1.0  2.0  3.0  4.0
# 2  NaN  NaN  3.0  4.0
# 3  1.0  NaN  NaN  4.0
# 4  1.0  2.0  NaN  NaN

And I would like to drop rows where all features in either sets {a1, a2} or {b1, b2} are NaN. So the output would be (rows number 2 and 4 dropped):

   a1   a2   b1   b2
0  1.0  NaN  3.0  4.0
1  1.0  2.0  3.0  4.0
3  1.0  NaN  NaN  4.0

I would ideally need some combination of df.dropna(how="all", subset=["a1", "a2"]) and df.dropna(how="all", subset=["b1", "b2"]) . In this simple case it would not be such problem but what about having eg 10 different subsets? (And in my real scenario it's almost 50.)

Is there any smart solution using pandas or any filter how to combine these subsets and make the right condition for dropna method?

Motivation : Just to give you an idea why I need something like this it's because I have different sets of features ( a , b , ...) that are combined in a single DataFrame and I need to handle those features separately. Some NaNs are ok, but if any row for any feature is full of NaNs, it means wrong measurement and I want to drop this row for any other feature as well (just imagine that the index is time of measurement and if a single set of features is incorrect I do not want to keep it even if other sets of features are fine).

Answer 1

Approach

For each subset in the list of predefined subsets you can test the columns of that subset for the presence of any non NaN value along axis=1 to create a boolean mask, then you can reduce all the boolean masks corresponding to each of the subset with np.logical_and to create a resulting boolean mask which then can be used to filter the rows in the dataframe.

subs = [{'a1', 'a2'}, {'b1', 'b2'}]
mask = np.logical_and.reduce([df[s].notna().any(1) for s in subs])

Result

>>> df[mask]

    a1   a2   b1   b2
0  1.0  NaN  3.0  4.0
1  1.0  2.0  3.0  4.0
3  1.0  NaN  NaN  4.0

Answer 2

First compare all values for not missing values, then group by first letter of columsnn ames with GroupBy.any for test if groups has no only NaNs and last filter by DataFrame.all for get all rows matching masks:

print (df.columns.str[0])
Index(['a', 'a', 'b', 'b'], dtype='object')


print (df.notna().groupby(df.columns.str[0], axis=1).any())
       a      b
0   True   True
1   True   True
2  False   True
3   True   True
4   True  False

df = df[df.notna().groupby(df.columns.str[0], axis=1).any().all(axis=1)]
print (df)
    a1   a2   b1   b2
0  1.0  NaN  3.0  4.0
1  1.0  2.0  3.0  4.0
3  1.0  NaN  NaN  4.0