How to merge columns and delete duplicates but keep unique values?

Question

I want to merge columns based on same IDs and want to make sure to consolidate the rows into just one row (per ID). Can anyone help me to merge the columns for duplicates and non-duplicates?

Given:

ID      Name     Degree       AM_Class     PM_Class     Online_Class
01      Kathy    Biology      Bio101       NaN          NaN
01      Kathy    Biology      NaN          Chem101      NaN
02      James    Chemistry    NaN          Chem101      NaN
03      Henry    Business     Bus100       NaN          NaN
03      Henry    Business     NaN          Math100      NaN
03      Henry    Business     NaN          NaN          Acct100

Expected Output:

ID      Name     Degree       AM_Class     PM_Class     Online_Class
01      Kathy    Biology      Bio101       Chem101      NaN
02      James    Chemistry    NaN          Chem101      NaN
03      Henry    Business     Bus100       Math100      Acct100

I tried to use:

df = df.groupby(['Name','Degree','ID'])['AM_Class', 'PM_Class', 'Online_Class'].apply(', '.join).reset_index()

but seems like it's giving an error..

Answer 1

Here is your data:

df = pd.DataFrame({'ID': ['01', '01', '02', '03', '03', '03'],
                   'Degree': ['Biology', 'Biology', 'Chemistry', 'Business', 'Business', 'Business'],
                   'Name': ['Kathy', 'Kathy', 'James', 'Henry', 'Henry', 'Henry'],
                   'AM_Class': ['Bio101', np.nan, np.nan, 'Bus100', np.nan, np.nan],
                   'PM_Class': [np.nan, 'Chem101', 'Chem101', np.nan, 'Math100', np.nan],
                   'Online_Class': [np.nan, np.nan, np.nan, np.nan, np.nan, 'Acct100']})

You can separate the data frames, remove the NaN values, then rejoin them.

The reduce() function allows the merge to be performed iteratively, without having to merge the data frames one by one.

from functools import reduce

# Separate the data frames
df_student = df[['ID', 'Name', 'Degree']]
df_AM = df[['ID', 'Name', 'AM_Class']]
df_PM = df[['ID', 'Name', 'PM_Class']]
df_OL = df[['ID', 'Name', 'Online_Class']]

# List of data frames
dfs = [df_student, df_AM, df_PM, df_OL]

# Remove all NaNs
for df in dfs:
    df.dropna(inplace=True)

# Merge dataframes without the NaNs
df_merged = reduce(lambda left, right: pd.merge(left, right, how='left', on=['ID', 'Name']), dfs)


    ID  Name    Degree      AM_Class    PM_Class    Online_Class
0   01  Kathy   Biology     Bio101      Chem101     NaN
1   01  Kathy   Biology     Bio101      Chem101     NaN
2   02  James   Chemistry   NaN         Chem101     NaN
3   03  Henry   Business    Bus100      Math100     Acct100
4   03  Henry   Business    Bus100      Math100     Acct100
5   03  Henry   Business    Bus100      Math100     Acct100

Then you just need to remove the duplicates.

df_merged.drop_duplicates(inplace=True).reset_index()

This is the result:

     ID Name    Degree      AM_Class    PM_Class    Online_Class
0    01 Kathy   Biology     Bio101      Chem101     NaN
1    02 James   Chemistry   NaN         Chem101     NaN
2    03 Henry   Business    Bus100      Math100     Acct100

Answer 2

You may ffill rows first and then drop duplicates while keeping the last occurrence of duplicates,

df.groupby(['ID']).ffill().drop_duplicates(subset='Name', keep='last')

Answer 3

we can use pandas pivot_table for this problem your data looks like this

>>> data = {'Name': ['Kathy','Kathy','James','Henry','Henry','Henry'],
        'Degree': ['Biology','Biology','Chemistry','Business','Business','Business'],
        'AM_Class': ['Bio101', np.nan, np.nan, 'Bus100', np.nan, np.nan],
        'PM_Class': [np.nan, 'Chem101', 'Chem101', np.nan, 'Math100', np.nan],
        'Online_Class': [np.nan, np.nan, np.nan, np.nan, np.nan, 'Acct100'],
        
       }
>>> df = pd.DataFrame(data)

>>> print(df)

 Name     Degree AM_Class PM_Class Online_Class
0  Kathy    Biology   Bio101      NaN          NaN
1  Kathy    Biology      NaN  Chem101          NaN
2  James  Chemistry      NaN  Chem101          NaN
3  Henry   Business   Bus100      NaN          NaN
4  Henry   Business      NaN  Math100          NaN
5  Henry   Business      NaN      NaN      Acct100

First we can replace all NaN with null string

>>> df.fillna('', inplace=True)

>>> print(df)

Name     Degree AM_Class PM_Class Online_Class
0     0    Biology   Bio101                      
1     1    Biology           Chem101             
2     2  Chemistry           Chem101             
3     3   Business   Bus100                      
4     4   Business           Math100             
5     5   Business                        Acct100

I am doing this because while using pivot_table function I would like to use np.sum function which will concatenate strings in the pandas.series. Having the np.nan as it is will raise exception.

Now lets make the pivot table with Name being the group-by column.

>>> df2 = pd.pivot_table(data=df, index=['Name'], aggfunc={'Degree':np.unique, 'AM_Class':np.sum, 'PM_Class':np.sum, 'Online_Class':np.sum})

>>> print(df2)

AM_Class     Degree Online_Class PM_Class
Name                                           
Henry   Bus100   Business      Acct100  Math100
James           Chemistry               Chem101
Kathy   Bio101    Biology               Chem101

We have to replace the nulls with np.nan - since that is the format that is asked for.

>>> df2.replace('', np.nan, inplace=True)

>>> print(df2)

AM_Class     Degree Online_Class PM_Class
Name                                           
Henry   Bus100   Business      Acct100  Math100
James      NaN  Chemistry          NaN  Chem101
Kathy   Bio101    Biology          NaN  Chem101

Observing the new dataframe df2 , it seems we have to make the following changes

• Since the name column has become the Index - we have to make a Name column
• Add a RangeIndex
• Column order has to be restored

>>> df2['Name'] = df2.index

>>> cols = [ 'Name', 'Degree', 'AM_Class',  'PM_Class', 'Online_Class']

>>> df2 = df2[cols]

>>> print(df2)

 Name     Degree AM_Class PM_Class Online_Class
Name                                                  
Henry  Henry   Business   Bus100  Math100      Acct100
James  James  Chemistry      NaN  Chem101          NaN
Kathy  Kathy    Biology   Bio101  Chem101          NaN

>>> df2.set_index(pd.RangeIndex(start=0,stop=3,step=1), inplace=True)

>>> print(df2)

 Name     Degree AM_Class PM_Class Online_Class
0  Henry   Business   Bus100  Math100      Acct100
1  James  Chemistry      NaN  Chem101          NaN
2  Kathy    Biology   Bio101  Chem101          NaN

Answer 4

If need first non missing values per groups use GroupBy.first :

df = df.groupby(['ID','Name','Degree'], as_index=False).first()
print (df)
   ID   Name     Degree AM_Class PM_Class Online_Class
0  01  Kathy    Biology   Bio101  Chem101         None
1  02  James  Chemistry     None  Chem101         None
2  03  Henry   Business   Bus100  Math100      Acct100

Or if need all unique values without missing values per groups use custom lambda function in GroupBy.agg for processing each column separately by Series.dropna , removed duplicated by dict.fromkeys and last join values by , :

f = lambda x: ', '.join(dict.fromkeys(x.dropna()))
df = df.groupby(['ID','Name','Degree'], as_index=False).agg(f).replace('', np.nan)

Difference is possible see in changed data:

print (df)
   ID   Name     Degree AM_Class PM_Class Online_Class
0  01  Kathy    Biology   Bio101      NaN          NaN
1  01  Kathy    Biology      NaN  Chem101          NaN
2  02  James  Chemistry      NaN  Chem101          NaN
3  03  Henry   Business   Bus100      NaN          NaN
4  03  Henry   Business      NaN  Math100      Acct100
5  03  Henry   Business      NaN  Math200      Acct100

df1 = df.groupby(['ID','Name','Degree'], as_index=False).first()
print (df1)
   ID   Name     Degree AM_Class PM_Class Online_Class
0  01  Kathy    Biology   Bio101  Chem101         None
1  02  James  Chemistry     None  Chem101         None
2  03  Henry   Business   Bus100  Math100      Acct100


f = lambda x: ', '.join(dict.fromkeys(x.dropna()))
df2 = df.groupby(['ID','Name','Degree'], as_index=False).agg(f).replace('', np.nan)
print (df2)
   ID   Name     Degree AM_Class          PM_Class Online_Class
0  01  Kathy    Biology   Bio101           Chem101          NaN
1  02  James  Chemistry      NaN           Chem101          NaN
2  03  Henry   Business   Bus100  Math100, Math200      Acct100