Python从CSV提取唯一值

Question

I am using the following python script to remove duplicates from a CSV file

with open('test.csv','r') as in_file, open('final.csv','w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    for line in in_file:
        if line in seen: continue # skip duplicate

        seen.add(line)
        out_file.write(line)

I am trying to modify it so that instead of outputting the list without duplicates to final.csv it outputs the unique values that were found.

Kind of the opposite to what it does now. Anyone got an example?

Answer 1

Using a dict to keep track of how many times each line occurs, then you can process the dict and add only the unique items to the seen set, and write those to the final.csv :

from collections import defaultdict
uniques = defaultdict(int)
with open('test.csv','r') as in_file, open('final.csv','w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    for line in in_file:
        uniques[line] +=1
    for k, v in uniques.iteritems():
        if v = 1:
            seen.add(k)
            out_file.write(k)

Or:

from collections import defaultdict
uniques = defaultdict(int)
with open('test.csv','r') as in_file, open('final.csv','w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    for line in in_file:
        uniques[line] +=1

    seen = set(k for k in uniques if uniques[k] == 1)
    for itm in seen:
        out_file.write(itm)

Or, using Counter :

from collections import Counter

with open('test.csv','r') as in_file, open('final.csv','w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    lines = Counter(file.readlines())
    seen = set(k for k in lines if lines[k] == 1)
    for itm in seen:
        out_file.write(itm)

This will output only the lines which appear once, depending on what you mean by "uniques", this may or may not be correct. If, instead, you want to output ALL lines but only one instance per line, using the last method:

with open('test.csv','r') as in_file, open('final.csv','w') as out_file:

    lines = Counter(file.readlines())

    for itm in lines:
        out_file.write(itm)

Answer 2

您可以将重复变量收集到另一个变量中，并使用它们从集合中删除非唯一值。