在 Pandas 中删除带有一部分列的 NaN 行

Question

I have hundreds of columns in a DataFrame and would like to drop rows where multiple columns are NaN. Meaning entire row is NaN for those columns.

I have tried to slice columns but the code is taking forever to run.

df = df.drop(df[(df.loc[:,'col1':'col100'].isna()) & (df.loc[:,'col120':'col220'].isna())].index)

Appreciate any help.

Answer 1

You should try to use the dropna() function with the subset parameter equal to the columns you are trying to drop on. Here is a short example taken from Pandas' documentation

df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
                   "toy": [np.nan, 'Batmobile', 'Bullwhip'],
                   "born": [pd.NaT, pd.Timestamp("1940-04-25"),
                            pd.NaT]})

df
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT

df.dropna(subset=['name', 'born'])

This gives you the following:

       name        toy       born
1    Batman  Batmobile 1940-04-25

Answer 2

Part of your original question reads: "... would like to drop rows where multiple columns are NaN. Meaning entire row is NaN for those columns. "

Can I interpret this as, you want to delete the row when the entire row has NaNs. If that is true you should be able to achive this by:

df.dropna(axis = 'rows', how = 'all', inplace = True)

If that is not the case then I misunderstood your question.