我认为这应该引发错误，但事实并非如此

Question

Below is a simple function to remove duplicates in a list while preserving order. I've tried it and it actually works, so the problem here is my understanding. It seems to me that the second time you run uniq.remove(item) for a given item, it will return an error ( KeyError or ValueError I think?) because that item has already been removed from the unique set. Is this not the case?

def unique(seq):
    uniq = set(seq)  
    return [item for item in seq if item in uniq and not uniq.remove(item)]

Answer 1

There's a check if item in uniq which gets executed before the item is removed. The and operator is nice in that it "short circuits". This means that if the condition on the left evaluates to False -like, then the condition on the right doesn't get evaluated -- We already know the expression can't be True -like.

Answer 2

set.remove is an in-place operation. This means that it does not return anything (well, it returns None ); and bool(None) is False .

So your list comprehension is effectively this:

answer = []
for item in seq:
    if item in uniq and not uniq.remove(item):
        answer.append(item)

and since python does short circuiting of conditionals (as others have pointed out), this is effectively:

answer = []
for item in seq:
    if item in uniq:
        if not uniq.remove(item):
            answer.append(item)

Of course, since unique.remove(item) returns None (the bool of which is False ), either both conditions are evaluated or neither.

The reason that the second condition exists is to remove item from uniq . This way, if/when you encounter item again (as a duplicate in seq ), it will not be found in uniq because it was deleted from uniq the last time it was found there.

Now, keep in mind, that this is fairly dangerous as conditions that modify variables are considered bad style (imagine debugging such a conditional when you aren't fully familiar with what it does). Conditionals should really not modify the variables they check. As such, they should only read the variables, not write to them as well.

Hope this helps

Answer 3

mgilson and others has answered this question nicely, as usual. I thought I might point out what is probably the canonical way of doing this in python, namely using the unique_everseen recipe from the recipe section of the itertools docs, quoted below:

from itertools import ifilterfalse

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in ifilterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

Answer 4

def unique_with_order(seq):
    final = []
    for item in seq:
        if item not in final:
            final.append(item)
    return final


print unique_with_order([1,2,3,3,4,3,6])

Break it down, make it simple :) Not everything has to be a list comprehension these days.

Answer 5

@mgilson's answer is the right one, but here, for your information, is a possible lazy ( generator ) version of the same function. This means it'll work for iterables that don't fit in memory - including infinite iterators - as long as the set of its elements will.

def unique(iterable):
    uniq = set()
    for item in iterable:
        if item not in uniq:
            uniq.add(item)
            yield item

Answer 6

The first time you run this function, you will get [1,2,3,4] from your list comprehension and the set uniq will be emptied. The second time you run this function, you will get [] because your set uniq will be empty. The reason you don't get any errors on the second run is that Python's and short circuits - it sees the first clause ( item in uniq ) is false and doesn't bother to run the second clause.