Python 加速 csv 操纵

Question

Is there anyway to speed up the processing of this csv file manipulation? With a csv with 5000 entries it works fine but when there are 1,000,000+ entries it takes a long time.

r1 = csv.reader(open('file1.csv'))
r2 = csv.reader(open('file2.csv'))
with open(file3, 'w', newline='') as wf:
    writer = csv.writer(wf)
    entries = []
    first = True

    for child, a, b, c, parent, d in r1:
        if not child and not parent:
            continue
        if first:
            first = False
            continue
        entries.append([parent, child])

    first = True

    for child, _, _, _, parent, _ in r2:
        if not child and not parent:
            continue
        if first:
            first = False
            continue

        entries.append([parent, child])

    for p, c in entries:
        for sp, sc in entries:
            if p == sc:
                break
        else:
            entries.append([p, p])


    writer.writerow(["parent_new", "child_new"])
    writer.writerows(entries)

Also there is a line break between the header and the first row of data, anyway to remove this blank line when writing to the new csv?

Answer 1

Your loop:

    for p, c in entries:
        for sp, sc in entries:
            if p == sc:
                break
        else:
            entries.append([p, p])

will be taking quadratic time.

All that it seems to be doing is writing the values of p which do not equal any of the child values. As these values derive from the CSV file so must be strings, and are therefore hashable, you could save them (or more specifically, the unique values) in a set:

children = set(child for parent, child in entries)

It costs some more memory, but then you can do

    for p, c in entries:
        if p not in children:
            entries.append([p, p])

so this should then be linear time rather than quadratic (because set inclusion testing is essentially constant time).

---

On a more minor point, to remove the first row of each of the input files, instead of using your first variable (which you then have to test for on every iteration), simply call next(r1) before entering the loop (and discard the value) -- and similarly for r2 . That said, do not expect huge gain from doing this, because this is in the linear-time part of the code. It is the O(n^2) bit mentioned above that is really important.