使用Python或php从大型CSV文件中计数唯一列值

Question

我有一个217gb的Csv文件，如何在没有超时的情况下使用python或php脚本获取唯一列值的计数？

Answer 1

Not sure what you mean by timeout, for big files like this it will always take a long time.

tokens = {}
with open("your.csv") as infile:
    for line in infile:
        columns = line.split(',')
        # Where idx is your desired column index
        if columns[idx] not in tokens:
            tokens[columns[idx]] = 0
        else:
            tokens[columns[idx]] += 1

print tokens

This loads the file line by line, so your compute doesn't crash from loading the whole 217 Gb into ram. You can try this first to see if the dictionary fits in your computer's memory. Otherwise you might wanna consider splitting the files to smaller chunks in a divide and conquer approach.

Answer 2

You could try to increase the field_size_limit

import csv
csv.field_size_limit(1000000000)

r = csv.reader(open('doc.csv', 'rb'))

for row in r:
    print(row)  # do the processing