How to remove duplicates from lists of lists

Question

During communities' detection I am trying to remove duplicates nodes from lists of lists (aimed to calculate ARI). What I have – few dozen lists inside one list with different dimensions:

lst_of_lts= [[5192, 32896, 34357, 34976, 36683, 43315], … ,[19, 92585, 94137, 98381, 99041, 100395, 101100, 109759]]

What I am running:

import itertools

Lst_of_lts.sort()

Lst_of_lts_2 = list(k for k,_ in itertools.groupby(Lst_of_lts))

Lst_of_lts_nodops= [list(i) for i in {tuple(sorted(i)) for i in Lst_of_lts_2}]

For some reason, it doesn't remove duplicates.

The dimensions remain the same- Any suggestions?

Also tried many options such as:

Remove duplicate items from lists in Python lists and Remove duplicated lists in list of lists in Python

Answer 1

If you are removing duplicates just in the list itself, you can use set.

a = np.random.randint(0,5,(10,10)).tolist()

a
Out[128]: 
[[0, 3, 0, 2, 4, 4, 0, 0, 3, 3],
 [2, 4, 0, 2, 4, 2, 2, 4, 3, 1],
 [3, 2, 0, 1, 2, 0, 2, 0, 2, 1],
 [3, 1, 4, 1, 0, 1, 4, 4, 3, 4],
 [2, 0, 1, 1, 0, 4, 1, 4, 2, 3],
 [0, 0, 1, 3, 4, 3, 1, 3, 0, 1],
 [1, 2, 0, 2, 1, 3, 4, 2, 2, 0],
 [3, 3, 2, 2, 0, 4, 1, 1, 0, 0],
 [0, 1, 3, 0, 4, 4, 2, 1, 1, 4],
 [0, 1, 4, 4, 0, 1, 3, 2, 1, 1]]

[list(set(i)) for i in a]
Out[129]: 
[[0, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3],
 [0, 1, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4]]

Or if you want to preserve the order of the element, you can use dict.fromkeys

[list(dict.fromkeys(i)) for i in a]
Out[133]: 
[[0, 3, 2, 4],
 [2, 4, 0, 3, 1],
 [3, 2, 0, 1],
 [3, 1, 4, 0],
 [2, 0, 1, 4, 3],
 [0, 1, 3, 4],
 [1, 2, 0, 3, 4],
 [3, 2, 0, 4, 1],
 [0, 1, 3, 4, 2],
 [0, 1, 4, 3, 2]]