解析一个巨大的 csv 文件，软件版本号有问题如何快速格式化 500 万行

Question

Here's a sample of my data:

from io import StringIO

data = StringIO("""software,version
Visual C++ Minimum Runtime,11.0.61030
Visual C++ Minimum Runtime,11.0.61030
Visual C++ Minimum Runtime,11.0.61030.0.0.0.0""")

Notice that the last record the version number has 0.0.0.0 in it.

How can I get to xx.yy.zz first front 3 characters and clean up the remaining data?

As an example: Visual C++ Minimum Runtime,11.0.61030.0.0.0.0 should be truncated to:

"Visual C++ Minimum Runtime,11.0.61030"

Is there an efficient way to accomplish this?

Answer 1

You could use generators to load the file row by row and then write the truncated rows to a backup file. eg.

import csv

filename = "foo.csv"

def get_row(filename):
    with open(filename, "rb") as csvfile:
        data = csv.reader(csvfile)
        yield next(data)

with open('truncated.csv','wb') as truncatedcsv:
    writer = csv.writer(truncatedcsv, delimiter=',')
    for row in get_row(filename):
        truncated_row = # your truncation logic
        writer.writerow(truncated_row)

Don't forget to rename the new file and delete the old one.