Python：从列表内部提取列表并删除重复项

Question

I have a dataframe with a column that consists of lists of lists (of varying length). One example: df['east'][0] gives

[array(['Indonesia', 'New Zealand'], dtype=object), array(['Indonesia', 'New Zealand'], dtype=object)]

I want to merge the lists inside of this bigger list and get rid of duplicates and make sure that data is presented clearly, ie simply

['Indonesia', 'New Zealand']

I tried some suggestions from here to remove duplicates, but, for example,for np.unique(functools.reduce(operator.add, east)) Python said "ValueError: operands could not be broadcast together with shapes (4,) (13,)"

I could usually solve problems, but here I am not sure what is happening - what are these arrays in the list.

Answer 1

One simple approach would be to flatten your lists/arrays with a comprehension and then use list(set()) to get unique values in a list:

df['east'].apply(lambda x: list(set(item for sublist in x for item in sublist)))
# example output: ['New Zealand', 'Indonesia']

Answer 2

you can use the following one liner to achieve your results.

df['east'].apply(lambda value: reduce(lambda a, x: list(set(list(a) + list(x))), value, []))

lets break it down...

list(a) + list(x) = avoids shape error and adds to lists to return one list (you can use addition of np arrays directly if you keep the shapes same)

list(set(list(a) + list(x))) = array of all unique elements by first taking their set.

reduce(lambda a, x: list(set(list(a) + list(x))), value, [])) = recursively adds accumulator and the variable list to reduce it into one single list.