从列表格式的数据框列中删除重复项

Question

I have ton of duplicate values in a data frame column by row. Below is some sample, I looked at other stack overflow question, but I can only find the answer for the list not for the data frame issue dupes. When I pass values in a list, I am able to remove duplicate values however, when I pass it like a data frame it is giving errors: TypeError: unhashable type: 'list'

What am I doing wrong here?

import pandas as pd 
d = {'col1': ['apples are delicious,apples are delicious,apples', 'apples'], 'col2': ['mangoes','oranges']}
df = pd.DataFrame(data=d)
df['col1'] = set(df['col1'].str.split(",")) #error tried list(set()) as well.
df['col2'] = df['col2'].str.split(",") #converting to list
print(df)

final output should remove dupes like this:

col1                                         co2
['apples are delicious','apples']            ['mangoes']
['apples']                                   ['oranges']

Answer 1

You are using set on an entire series, whereas you need to apply set to each element in the series. For this, you can use pd.Series.map :

df['col1'] = df['col1'].str.split(',').map(set)

print(df)

                             col1       col2
0  {apples are delicious, apples}  [mangoes]
1                        {apples}  [oranges]

Your error derives from the fact you can't have a set of lists since lists are not hashable.

If you really need a series of lists as the result, you can use the same method again, ie df['col1'].str.split(',').map(set).map(list) . But note that you should assume no ordering as set is an unordered collection.