If I have duplicates in a list with brackets, what should I do

Question

Suppose I have the following list:

 m=[1,2,[1],1,2,[1]]

I wish to take away all duplicates. If it were not for the brackets inside the the list, then I could use:

 m=list(set(m))

but when I do this, I get the error:

unhashable type 'set'.

What command will help me remove duplicates so that I could only be left with the list

 m=[1,2,[1]]

Thank you

Answer 1

You can do something along these lines:

m=[1,2,[1],1,2,[1]]
seen=set()
nm=[]
for e in m:
    try:
        x={e}
        x=e
    except TypeError:
        x=frozenset(e)  
    if x not in seen:
        seen.add(x) 
        nm.append(e)
>>> nm
[1, 2, [1]]

---

From comments: This method preserves the order of the original list. If you want the numeric types in order first and the other types second, you can do:

sorted(nm, key=lambda e: 0 if isinstance(e, (int,float)) else 1)

Answer 2

result = []
for i in m:
  flag = True
  for j in m:
    if i == j:
      flag = False
  if flag:
    result.append(i)

Result will be: [1,2,[1]]

There are ways to make this code shorter, but I'm writing it more verbosely for readability. Also, note that this method is O(n^2), so I wouldn't recommend for long lists. But benefits is the simplicity.

Answer 3

The first step will be to convert the inner lists to tuples:

>> new_list = [tuple(i) if type(i) == list else i for i in m]

Then create a set to remove duplicates:

>> no_duplicates = set(new_list)
>> no_duplicates
{1, 2, (1,)}

and you can convert that into list if you wish.

Answer 4

For a more generic solution you can serialize each list item with pickle.dumps before passing them to set() , and then de-serialize the items with pickle.loads :

import pickle
m = list(map(pickle.loads, set(map(pickle.dumps, m))))

If you want the original order to be maintained, you can use a dict (which has become ordered since Python 3.6+) instead of a set:

import pickle
m = list(map(pickle.loads, {k: 1 for k in map(pickle.dumps, m)}))

Or if you need to be compatible with Python 3.5 or earlier versions, you can use collections.OrderedDict instead:

import pickle
from collections import OrderedDict
m = list(map(pickle.loads, OrderedDict((k, 1) for k in map(pickle.dumps, m))))

Answer 5

Simple Solution,

m=[1,2,[1],1,2,[1]]
l= []
for i in m:
    if i not in l:
        l.append(i)
print(l)

[1, 2, [1]]

[Program finished]