在python中使用for循环处理时写出每一行，但只写第一行

Question

I have some code that I am trying to optimize to be more efficient. One part of that is to process my file and after each line is processed, immediately write it to csv . This is ideal as then I am not wasting memory by processing the data, then loading the data into a list to write out the entire list. If I add the entire processed data to a list, I can write it to csv without trouble, as shown below, under # write folded_data to csv :

Note: the code under #data processing is solid, and I just need help writing out each row as it's processed.

# data processing
seen = set()
folded_data = []
for u in name_nodes:
#    seen=set([u]) # print both u-v, and v-u
    seen.add(u) # don't print v-u
    unbrs = set(B[u])
    nbrs2 = set((n for nbr in unbrs for n in B[nbr])) - seen
    for v in nbrs2:
        vnbrs = set(B[v])
        common = unbrs & vnbrs
        weight = len(common)
        row = u, v, weight
        folded_data.append(row)

# write folded_data to csv
with ('out_file.csv', 'wb') as f:
    writer = csv.writer(f)
    writer.writerows(folded_data)

However, when I try and write out each row as it's processed, I only get the first line in 'out_file.csv'.

# data processing
seen = set()
for u in name_nodes:
    # seen=set([u]) # print both u-v, and v-u
    seen.add(u) # don't print v-u
    unbrs = set(B[u])   
    nbrs2 = set((n for nbr in unbrs for n in B[nbr])) - seen
    for v in nbrs2:
        vnbrs = set(B[v])
        common = unbrs & vnbrs
        weight = len(common)
        row = u, v, weight
        # write row for each line to csv
        with open('out_file.csv', 'wb') as f:
            writer = csv.writer(f)
            writer.writerow(row)

I've tried moving my writing code around to make this work as I would like it to, but I haven't been able to figure this out.

Answer 1

I doubt that you're getting the first line, you're getting the last line. For each line you write out, you're reopening the file, erasing the previous contents. Put the file open and csv writer creation outside of the loop.

Answer 2

I wouldn't worry about "wasting" memory unless your program asks for (eg) greater than 1/2 of the system memory. If your CSV is in the multi-gigabyte range (or bigger) then this is a concern.

If your csv is not this large, your file will end up in the OS file cache in memory, unless you have some non-standard kernel settings.

To do it the "efficient" way (ie to not explicitly store your data in memory), you need to open the file before the for loop.

Answer 3

Figured it out with the help of @etep and @MarkRansom! I have to open the file and define writer before the entire for-loop .

# open file and define writer
with open('out_file.csv', 'wb') as f:
    writer = csv.writer(f)

    # data processing
    seen = set()
    for u in name_nodes:
    #    seen=set([u]) # print both u-v, and v-u
        seen.add(u) # don't print v-u
        unbrs = set(B[u])
        nbrs2 = set((n for nbr in unbrs for n in B[nbr])) - seen
        for v in nbrs2:
            vnbrs = set(B[v])
            common = unbrs & vnbrs
            weight = len(common)
            row = u, v, weight
            # write row for each record
            writer.writerow(row)