根据pandas中的特定条件删除特定行

Question

I want to clean my data, basically I have my data -

dataframe -

d = {'User': ['Mansi kinney', 'Mansi kinney', 'Mansi kinney', 'Alley Huff', 'Alley Huff', 'Alley Huff',  Raedden Grip', 'Raedden Grip',  'S.Sarkar',
                              'S.Sarkar', 'S.Sarkar'],
                      'Work': ['', '', '', 'College', 'College', 'College', '', '', 'Business', 'Business', 'Business'],
                      'Country': ['Aus', 'Aus', 'Australia', 'US','US', 'US', 'Ban', 'Ban',
                                 'Ind', 'Ind', 'Ind'],
                      'Dept': ['Safety', 'Safety', 'Safety', '', '', '', '', '', '', '', ''],
                      'Training': ['', 'Internal', '', '', 'External', '', '', '', '', 'Internal', ''],
                      'Status': ['', '', 'Active', '', '', 'Active', '', 'Active', '', '', '']
        }
    df = pd.DataFrame(data=d)
    df

Here I want to delete the rows where the more no. Of cells are blank and the data is scattered so I want it to be in a single row and delete the unnecessary repitition of rows

My output should be-

d = {'User':['Mansi kinney','Alley Huff','Raedden Grip', 'S.Sarkar'],
'Work': ['', 'College', '', 'Business'],
'Country': ['Aus', 'US', 'Ban',  'Ind'],
'Dept': ['Safety', '', '', ''],
                      'Training':['Internal','External', '', 'Internal'],
'Status':['Active','Active','Active', 'Active']
        }
    df = pd.DataFrame(data=d)
    df

I have typed the whole thing in my smartphone so please let me know if the question is clear or not. Please help me to clean the data and get the desired output. Thanks in advance!!!

Answer 1

You can groupby 'User' and aggregate with ' '.join and remove duplicates with unique():

df = df.groupby('User').agg(lambda x: ''.join(x.unique()))
df.reset_index(inplace=True)

print(df)  
#output:
           User      Work       Country    Dept  Training  Status
0    Alley Huff   College            US          External  Active
1  Mansi kinney            AusAustralia  Safety  Internal  Active
2  Raedden Grip                     Ban                    Active
3      S.Sarkar  Business           Ind          Internal  

UPDATE: Here it is with your full data

import pandas as pd
df = pd.read_excel('your_data.xls')
df = df.groupby('user').agg(lambda x: ''.join(x.unique()))
df.reset_index(inplace=True)
pd.set_option('display.max_columns', None)
print(df)

#output:
                   user                                              works  \
0        Abhishek Mitra  Director & CEO | INDIAN CYBER SECURITY SOLUTIO...   
1         Anandita Kaul       HR Associate - Recruitments at Pulp Strategy   
2         Glam Sorvey.B    Data Science & Analytics with Python Consultant   
3   Madhurima S. Sarkar  MBA in Business Analytics and Finance || The S...   
4         Mansi Makhija                                    Works at Amazon   
5       NanaoSana Singh  DevOps | AWS | Docker | Kubernetes | Git | Jen...   
6         Neeraj Mishra                               DGM at Pulp Strategy   
7       Niral Shahpatel  Scale and Strategy Lead - Global Partner Marke...   
8      Sandhya Ramagiri     Technical Program Manager at Intel Corporation   
9         Sarthak Ahuja         Associate Account Manager at Pulp Strategy   
10         UDIT NARAYAN          Student at Narula Institute Of Technology   

                             country  
0        Kolkata, West Bengal, India  
1          South Delhi, Delhi, India  
2    Portland, Oregon, United States  
3        Kolkata, West Bengal, India  
4        Noida, Uttar Pradesh, India  
5           Pune, Maharashtra, India  
6                              India  
7   Hillsboro, Oregon, United States  
8    Austin, Texas Metropolitan Area  
9                       Delhi, India  
10       Domchanch, Jharkhand, India