从python中的元组列表中删除唯一的元组

Question

I'm writing a program that find duplicated files, and right now I have a list of tuples as

mylist = [(file1, size1, hash1),
          (file2, size2, hash2),
          ...
          (fileN, sizeN, hashN)]

I want to remove the entries that have a unique hash, leaving only the duplicates. I'm using

def dropunique(mylist):
templist = []
while mylist:
    mycandidate = mylist.pop()
    templist.append([mycandidate])
    for myfile in mylist:
        if myfile[-1] == mycandidate[-1]:
            templist[-1].append(myfile)
            mylist.remove(myfile)
for myfile in templist:
    if len(myfile) != 1:
        mylist.append(myfile)
templist = [item for sublist in mylist for item in sublist]
return templist

where I pop an entry, look if there is other one with the same hash and group then in a list of list with the same hash. Then I make another list just with the sublists with len > 1 and flat the resulting list of lists into a simple list. My problem is that when I remove an entry from a list while using 'for myfile in mylist:' on the some list, it jumps same entries and live then behind.

Answer 1

Copy your list in a dictionary where the hash is the key, and on a second pass remove those with a single count - you can even use collections.Counter to spare one or two lines of code:

from collections import Counter

counter = Counter(t[2] for t in list_)

result = [value for value in list_ if counter[value[2]] > 1]

(Non-related tip: avoid naming your variables as "list" or "dict" - that overrides Python default built-ins for those)

Answer 2

I would use a defaultdict() to group the tuples by their hashvalue:

from collections import defaultdict

# Group the tuples by their hashvalues
d = defaultdict(list)
for tup in data:
    filename, size, hashvalue = tup
    d[hash].append(tup)

# Display groups of tuples that have more than one tuple
for hashvalue, tuples in d.items():
    if len(tuples) > 1:
        print('Tuples with %r in common' % hashvalue)
        for tup in tuples:
            print(tup)
        print()

Answer 3

Solution using groupby

from itertools import groupby

my_list = [(1, 2, 3),
           (1, 2, 3),
           (4, 5, 6)]


vals = []

for hash_val, items in groupby(sorted(my_list), hash):
    results = tuple(items)
    if len(results) > 1:
        vals.append(results[0])

Answer 4

You can use double filter like this:

filter(lambda el: len(filter(lambda item: item[2] == el[2], my_list)) > 1, my_list)

Result:

>>> my_list = [('file1', 'size1', 'hash1'), ('file2', 'size2', 'hash2'), ('file3', 'size3', 'hash3'), ('file4', 'size4', 'hash2')]
>>>
>>> filter(lambda el: len(filter(lambda item: item[2] == el[2], my_list)) > 1, my_list)
[('file2', 'size2', 'hash2'), ('file4', 'size4', 'hash2')]

Note that in Python 3, filter returns an iterator, so you'll need to convert it to a list like this: list(filter(...))