Most efficient way of keeping only the unique items in a list?

Question

So I'm relatively new to Python and trying to figure out what's the best way to keep only unique items in a list. My current implementation involves a Counter, dict and list comprehensions, but I'm not sure what may be faster.

Here's an example of what I've tried:

l = ['a', 'b', 'a']
d = dict(Counter(l))
[key for key, val in d.items() if val == 1]
>>> ['b']

Also, this only works for strings and not ints and I'm not sure why.

Answer 1

Do you want only things that exist one time?

>>> c=Counter(['a','b','a'])
>>> [n for n in c if c[n]==1]
['b']
>>> c=Counter([1,2,3,2,3,4,5,6,5,6])
>>> [n for n in c if c[n]==1]
[1, 4]

Or just a list of unique things?

>>> set([1,2,3,2,3,4,5,6,5,6])
set([1, 2, 3, 4, 5, 6])

Answer 2

Python has a built in type for ensuring that the members in a list are unique, it's a set . Using your example:

l = ['a', 'b', 'a']
set(l)
>>> ['a','b']

Commonly, you can "wash" the duplicate members from a list by converting from a list, to a set, and back again. For example:

l = ['a', 'b', 'a']
list(set(l))
>>> ['a','b']

This will turn the list back into a mutable (editable) list and ensures the best combination of performance and convenience.

Answer 3

Nothing wrong with the way you were doing it. Though the dict is superflurous. This is quite efficient but will only work if the "keys" are all hashable

[k for k,v in Counter(L).iteritems() if v==1]

Answer 4

If you want to remove duplicate items, use a set , then re-convert the result to a list:

ls = [1, 2, 3, 3, 3, 'a', 'b', 'b', 'c']
unique = list(set(ls))
# unique is ['a', 1, 2, 3, 'c', 'b']

Note that this operation won't preserve the order of the elements.

Answer 5

If you don't care about the order, just use set() . However the following will preserve the order:

l = ['a', 'b', 'c', 'a', 'c', 'd']

a = []
for item in l:
   if item not in a: a.append(item)

Or to only keep unique items:

l = [item for item in l if l.count(item) == 1]