Removing duplicates from list of lists by using list comprehension

Question

I was curious if you could remove duplicates from list of lists and return uniques as a list. I was trying this:

def do_list( lists ):
    res = [ [ one for one in temp if one not in res ] for temp in lists ]
    return res

So for example, if:

lists = [ [ "a","b","c" ],[ "d","a" ],[ "c","a","f" ] ]

the result should be:

[ "a","b,"c","d","f" ]    

But it gives me error that I refference variable res before assigment.

Answer 1

You could do this:

set(itertools.chain.from_iterable(lists))

set will remove all duplicates, inside of the set is just flattening your list(s) to a single list.

Answer 2

You get an error because you're referencing res inside the comprehension. This doesn't work, since res is only available after the expression is finished.

As I'm a curious sort, and because the title asks "Removing duplicates from list of lists by using list comprehension", I wanted to see if you can do this using only a list comprehension, and not by cheating such as using itertools :p

And here's how:

>>> lists = [ [ "a","b","c" ],[ "d","a" ],[ "c","a","f" ] ]
>>> lists2 = sorted(sum(lists, []))
>>> [ item for i, item in enumerate(lists2) if i == 0 or i == len(lists2) or lists2[i - 1] != item ]
['a', 'b', 'c', 'd', 'f']

For more insanity, you can combine them on a single line, but you'd have to repeat the sum() and sorted() calls. I couldn't move myself to write such ugly code ;-)

• sum(lists, []) will flatten the list; it returns the sum ( + operator) of all the items in lists , with [] as the initial list.
• sorted() will sort it. This is needed since we only check against the last item
• the if statement checks if the previous item is the same as the current item.

But it's ugly and un-Pythonic. For the love of Guido, use Pythonista's answer (or some variation thereof)!

Answer 3

res isn't created until the entire list comprehension has been evaluated. You can use a set to remove duplicates:

res = list(set(sum(lists, [])))

If you want it sorted:

res = sorted(set(sum(lists, [])))

If you want it ordered exactly how it comes, a list comprehension is probably not the best way. Instead, do this:

res = []
for temp in lists:
    res.append([])
    for one in temp:
        if one not in res[-1]:
            res[-1].append(one)