仅从数据框中的一行中删除 Nan 值

Question

I have a dataframe which looks something like this:

Df
lev1    lev2   lev3    lev4   lev5   description
RD21    Nan    Nan     Nan    Nan    Oil
Nan     RD32   Nan     Nan    Nan    Oil/Canola
Nan     Nan    RD33    Nan    Nan    Oil/Canola/Wheat
Nan     Nan    RD34    Nan    Nan    Oil/Canola/Flour
Nan     Nan    Nan     RD55   Nan    Oil/Canola/Flour/Thick
ED54    Nan    Nan     Nan    Nan    Rice
Nan     ED66   Nan     Nan    Nan    Rice/White
Nan     Nan    ED88    Nan    Nan    Rice/White/Jasmine
Nan     Nan    ED89    Nan    Nan    Rice/White/Basmati
Nan     ED68   Nan     Nan    Nan    Rice/Brown

I want to remove all the NaN values and just keep the non Nan values, something like this:

DF2
code     description
RD21     Oil
RD32     Oil/Canola
RD33     Oil/Canola/Wheat
RD34     Oil/Canola/Flour
RD55     Oil/Canola/Flour/Thick
.
.
.

How do I do this? I tried using notna() method, but it returns a boolean value of the dataframe. Any help would be appreciated.

Answer 1

You can use stack and groupby like this to find the fist non null value,

df['code'] = df[['lev1', 'lev2', 'lev3', 'lev4', 'lev5']].stack().groupby(level=0).first().reindex(df.index)

Now, you can select the code column and description column

df[['code', 'description']]


   code             description
0  RD21                     Oil
1  RD32              Oil/Canola
2  RD33        Oil/Canola/Wheat
3  RD34        Oil/Canola/Flour
4  RD55  Oil/Canola/Flour/Thick
5  ED54                    Rice
6  ED66              Rice/White
7  ED88      Rice/White/Jasmine
8  ED89      Rice/White/Basmati
9  ED68              Rice/Brown

Answer 2

We can mask by notna()

import pandas as pd
import numpy as np

df1 = pd.DataFrame(
    {
        'l1': [np.nan, 5],
        'l2': [6, np.nan],
        'd': ['a', 'b']
     }
)

notna = df1[['l1', 'l2']].notna().values
notna_values = df1[['l1', 'l2']].values[notna]
print(notna_values)

df2 = pd.DataFrame(df1['d'])
df2['code'] = notna_values

print(df2)

out:

   d  code
0  a   6.0
1  b   5.0

Answer 3

You can apply a function over every row in df[cols] (the subview over problematic columns), dropping every NaN and taking the only one remaining.

>>> cols = "lev1    lev2   lev3    lev4   lev5".split()
>>> df["code"] = df[cols].apply(lambda row: row.dropna().iloc[0])

You can also drop the original columns if you don't need them anymore with drop .

>>> df.drop(columns=cols, inplace=True)

Answer 4

Select columns like lev and replace NaN with "back fill". Keep the first level lev1 and concat to your other column(s).

>>> pd.concat([df.filter(like='lev').bfill(axis='columns')['lev1'].rename('code'),
               df['description']], axis="columns")

   code             description
0  RD21                     Oil
1  RD32              Oil/Canola
2  RD33        Oil/Canola/Wheat
3  RD34        Oil/Canola/Flour
4  RD55  Oil/Canola/Flour/Thick
5  ED54                    Rice
6  ED66              Rice/White
7  ED88      Rice/White/Jasmine
8  ED89      Rice/White/Basmati
9  ED68              Rice/Brown

or using melt :

>>> df.melt('description', value_name='code') \
      .dropna().drop(columns='variable') \
      .reset_index(drop=True) \
      [['code', 'description']]

   code             description
0  RD21                     Oil
1  ED54                    Rice
2  RD32              Oil/Canola
3  ED66              Rice/White
4  ED68              Rice/Brown
5  RD33        Oil/Canola/Wheat
6  RD34        Oil/Canola/Flour
7  ED88      Rice/White/Jasmine
8  ED89      Rice/White/Basmati
9  RD55  Oil/Canola/Flour/Thick

Answer 5

With your dataframe, I'd first make sure that Nan is actual np.NaN and not a string saying 'Nan'. Then I'd want to make sure that they're imputed as empty strings. Thus,

df.replace('Nan', np.nan, inplace=True)
df.fillna('', inplace=True)

Afterwards,

df['code'] = df['lev1'] + df['lev2'] + df['lev3'] + df['lev4'] + df['lev5']

And then df.drop(columns=[s for s in df.columns if s.startswith('lev')], inplace=True) to dispose of the old columns.

Note that this works only with the assumption given in OP's comment that there is one unique code in the five columns and the others are all NaN .