How to remove duplicates in this list?

Question

I am trying to remove the duplicates in the following list

[[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}], 
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}], 
 [{4, 5, 7}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}], 
 [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}], 
 [{6, 7, 8, 9, 10}, {1, 2, 3, 4, 5}]]

my expected output is

[[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}], 
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],  
 [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}]]

anyone knows any methods to solve the problem?

Answer 1

You could do:

import pprint

data = [[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}],
        [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],
        [{4, 5, 7}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}],
        [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}],
        [{6, 7, 8, 9, 10}, {1, 2, 3, 4, 5}]]

# find the uniques, keep order of appearance
uniques = dict.fromkeys([frozenset(frozenset(s) for s in e) for e in data])

# transform to original format
res = [[set(s) for s in e] for e in uniques]

pprint.pprint(res)

Output

[[{1, 2, 3, 8, 9, 10}, {5, 6, 7}, {4, 5, 7}],
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],
 [{1, 2, 3, 4, 5}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}]]

The frozenset is a hashable version of a set. The function dict.fromkeys , keeps the order of appearance in Python 3.6+.