如何在实例列表中删除“重复项”

Question

I have a list of instances of a certain class. This list contains `duplicates', in the sense that duplicates share the exact same attributes. I want to remove the duplicates from this list.

I can check whether two instances share the same attributes by using

class MyClass:

    def __eq__(self, other) : 
        return self.__dict__ == other.__dict__

I could of course iterate through the whole list of instances and compare them element by element to remove duplicates, but I was wondering if there is a more pythonic way to do this, preferably using the in operator + list comprehension.

Answer 1

set s (no order)

A set cannot contain duplicate elements. list(set(content)) will deduplicate a list. This is not too inefficient and is probably one of the better ways to do it :P You will need to define a __hash__ function for your class though, which must be the same for equal elements and different for unequal elements for this to work. Note that the hash value must obey the aforementioned rule but otherwise it may change between runs without causing issues.

index function (stable order)

You could do lambda l: [l[index] for index in range(len(l)) if index == l.index(l[index])] . This only keeps elements that are the first in the list.

in operator (stable order)

def uniquify(content):
    result = []
    for element in content:
        if element not in result:
            result.append(element)
    return result

This will keep appending elements to the output list unless they are already in the output list.

Answer 2

A little more on the set approach. You can safely implement a hash by delegating to a tuple's hash - just hash a tuple of all the attributes you want to look at. You will also need to define an __eq__ that behaves properly.

class MyClass:
    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c

    def __eq__(self, other):
        return (self.a, self.b, self.c) == (other.a, other.b, other.c)

    def __hash__(self):
        return hash((self.a, self.b, self.c))

    def __repr__(self):
        return "MyClass({!r}, {!r}, {!r})".format(self.a, self.b, self.c)

As you're doing so much tuple construction, you could just make your class iterable:

def __iter__(self):
    return iter((self.a, self.b, self.c))

This enables you to call tuple on self instead of laboriously doing .a, .b, .c etc.

You can then do something like this:

def unordered_elim(l):
    return list(set(l))

If you want to preserve ordering, you can use an OrderedDict instead:

from collections import OrderedDict

def ordered_elim(l):
    return list(OrderedDict.fromkeys(l).keys())

This should be faster than using in or index , while still preserving ordering. You can test it something like this:

data = [MyClass("this", "is a", "duplicate"),
        MyClass("first", "unique", "datum"),
        MyClass("this", "is a", "duplicate"),
        MyClass("second", "unique", "datum")]

print(unordered_elim(data))
print(ordered_elim(data))

With this output:

[MyClass('first', 'unique', 'datum'), MyClass('second', 'unique', 'datum'), MyClass('this', 'is a', 'duplicate')]
[MyClass('this', 'is a', 'duplicate'), MyClass('first', 'unique', 'datum'), MyClass('second', 'unique', 'datum')]

NB if any of your attributes aren't hashable, this won't work, and you'll either need to work around it (change a list to a tuple) or use a slow, n ^ 2 approach like in .