从CSV文件中提取没有重复的列表

Question

I have a dataset which looks like this:

id,created_at,username
1,2006-10-09T18:21:51Z,hey
2,2007-10-09T18:30:28Z,bob
3,2008-10-09T18:40:33Z,bob
4,2009-10-09T18:47:42Z,john
5,2010-10-09T18:51:04Z,brad
...

I contains 1M+ lines. I'd like to extract the list of username without duplicate from it using python. So far my code looks like this:

import csv

file1 = file("sample.csv", 'r')
file2 = file("users.csv", 'w')

reader = csv.reader(file1)
writer = csv.writer(file2)

rownum = 0
L = []
for row in reader:
    if not rownum == 0:
        if not row[2] in L:
            L.append(row[2])
            writer.writerow(row[2])

    rownum += 1

I have several questions: 1 - my output in users.csv looks like this:

h,e,y
b,o,b
j,o,h,n
b,r,a,d

How do I remove the commas between each letter?

2 - My code is not very elegant, is there any way to import the csv file as a matrix to select the last row and then to use an elegant library like underscore.js in javascript to remove the duplicates?

Many thanks

Answer 1

You can use a set here, it provides O(1) item lookup compared to O(N) of lists.

seen = set()
add_  = seen.add
next(reader) #skip header
writer.writerows([row[-1]] for row in reader if row[-1] not in seen
                                                        and not add_(row[-1]))

And always use the with statement for handling files, it'll automatically close the file for you:

with file("sample.csv", 'r') as file1, file("users.csv", 'w') as file2:
    #Do stuff with file1 and file2 here

Answer 2

Change

writer.writerow(row[2])

to

writer.writerow([row[2]])

Also, checking for membership in lists is computationally expensive [O(n)]. If you will be checking for membership in a large collection of items, and doing it often, use a set [O(1)]:

L = set()
reader.next() # Skip the header
for row in reader:
    if row[2] not in L:
        L.add(row[2])
        writer.writerow([row[2]])

Alternatively

If you're okay with using a few megabytes of memory, just do this:

with open("sample.csv", "rb") as infile:
    reader = csv.reader(infile)
    reader.next()
    no_duplicates = set(tuple(row) for row in reader)

    with open("users.csv", "wb") as outfile:
        csv.writer(outfile).writerows(no_duplicates)

if order is important, use an OrderedDict instead of a set:

from collections import OrderedDict
with open("sample.csv", "rb") as infile:
    reader = csv.reader(infile)
    reader.next()
    no_duplicates = OrderedDict.fromkeys(tuple(row) for row in reader)

    with open("users.csv", "wb") as outfile:
        csv.writer(outfile).writerows(no_duplicates.keys())

Answer 3

Easy and short!

for line in reader:
    string = str(line)
    split = string.split("," , 2)
    username = split[2][2:-2]