csv.writerow 似乎无法写入行

Question

I have a script that writes close to 800,000 rows to various CSV files.

for col in data:
        with open('output/{}.csv'.format(col), mode, encoding='utf-8', newline='') as f:
            writer = csv.writer(f, lineterminator = '\n')
            if mode == 'w':
                writer.writerow(headers)
            for row in data[col]:
                writer.writerow(row)

On modern machines this appears to function correctly and write all rows.

However on older machines (Using mechanical hard drives) there is around 35% to 40% rows missing in total.

Here is a list of the machine, and the total rows written (out of 815143):

• Macbook pro osx 10.15(16gb ram, SSD) - 815143
• Windows 10 (32gb ram, NVme) - 815143
• Windows 10 (4gb ram, HDD) - 543737
• Windows 10 (4gb ram, HDD) - 501335

Is there something I am doing incorrectly that is causing this?

Or are write failures something to consider? when using csv.write ?

Answer 1

It appears that the cause of this issue is Python's output buffer.

https://blog.finxter.com/what-is-python-output-buffering-and-how-to-disable-it/

passing the -u flag when running python appears to fix the issue of csv rows not being written on slower machines.

Eg:

python -u file.py argument

Hopefully this helps anybody else who encounters the same issue.