Compare 2 list columns in a pandas dataframe. Remove value from one list if present in another

Question

Say I have 2 list columns like below:

group1 = [['John', 'Mark'], ['Ben', 'Johnny'], ['Sarah', 'Daniel']]
group2 = [['Aya', 'Boa'], ['Mab', 'Johnny'], ['Sarah', 'Peter']]

df = pd.DataFrame({'group1':group1, 'group2':group2})

I want to compare the two list columns and remove the list elements from group1 if they are present in group2 . So expected results for above:

    group1                       group2
['John', 'Mark']             ['Aya', 'Boa']
['Ben']                     ['Mab', 'Johnny']
['Daniel']                  ['Sarah', 'Peter']

How can I do this? I have tried this:

df['group1'] = [[name for name in df['group1'] if name not in df['group2']]]

But got errror:

TypeError: unhashable type: 'list'

Please help.

Answer 1

You need to zip the two Series. I'm using a set here for efficiency (this is not critical if you have only a few items per list):

df['group1'] = [[x for x in a if x not in S]
                for a, S in zip(df['group1'], df['group2'].apply(set))]

Output:

         group1          group2
0  [John, Mark]      [Aya, Boa]
1         [Ben]   [Mab, Johnny]
2      [Daniel]  [Sarah, Peter]

Answer 2

You can use set difference:

df.apply(lambda x: set(x['group1']).difference(x['group2']), axis=1)

Output:

0    {John, Mark}
1           {Ben}
2        {Daniel}
dtype: object

To get lists you can add .apply(list) at the end.

Answer 3

you can use a loop in lambda function:

df['group1']=df[['group1','group2']].apply(lambda x: [i for i in x['group1'] if i not in x['group2']],axis=1)
print(df)
'''
         group1          group2
0  [John, Mark]      [Aya, Boa]
1         [Ben]   [Mab, Johnny]
2      [Daniel]  [Sarah, Peter]
'''