Pandas 忽略 NaN 删除重复项

Question

In a Pandas df, I am trying to drop duplicates across multiple columns. Lots of data per row is NaN .

This is only an example, the data is a mixed bag, so many different combinations exist.

df.drop_duplicates()

    IDnum       name            formNumber
1   NaN         AP GROUP        028-11964
2   1364615.0   AP GROUP        NaN
3   NaN         AP GROUP        NaN

Hopeful Output:

    IDnum       name            formNumber
1   1364615.0   AP GROUP        028-11964

EDIT:

If the df.drop_duplicates() looks like this, would it change the solution? :

df.drop_duplicates()

    IDnum       name            formNumber
0   NaN         AP GROUP        028-11964
1   1364615.0   AP GROUP        028-11964
2   1364615.0   AP GROUP        NaN
3   NaN         AP GROUP        NaN

Answer 1

You can using groupby + first

df.groupby('name',as_index=False).first()
Out[206]: 
      name      IDnum formNumber
0  APGROUP  1364615.0  028-11964

Answer 2

You need:

df.bfill().ffill().drop_duplicates()

Output:

IDnum   name    formNumber
0   1364615.0   AP GROUP    028-11964

Answer 3

There are multiple ways we can remove duplicates from a dataframe. few common ways are:

#option 1
df.drop_duplicates()

#option 2
df.groupby(df.columns.tolist()).size()

The major difference between this two options are:

1. option 1 considers NAN values. for example in your case

    df.drop_duplicates() IDnum name formNumber 0 NaN AP GROUP 028-11964 1 1364615.0 AP GROUP 028-11964 2 1364615.0 AP GROUP NaN 3 NaN AP GROUP NaN

Here index 0,1,2,3 all are unique rows although duplicates exist in some form.

2. option 2 considers only non NAN values and filters the duplicates as explained in first answer by @BENY

df.groupby('name',as_index=False).first()

      name    IDnum formNumber
0  APGROUP  1364615.0  028-11964

here in the above case we see only one unique and non duplicated value as group by did not consider NAN's.

To better understand this we can do:

df.drop_duplicates().info()
df.groupby(df.columns.tolist(),as_index=False).first().info()

by running the above code we get different count for "non-null" records. this explains how many nulls ignored in 2nd option as compared to 1st option.