用Python方式将大量行写入文件

Question

I need to auto-generate a somewhat large Makefile using a Python script. The number of lines is expected to be relatively large. The routine for writing to the file is composed of nested loops, whole bunch of conditions etc.

My options:

1. Start with an empty string and keep appending the lines to it and finally write the huge string to the file using file.write (pro: only single write operation, con: huge string will take up memory)

2. Start with an empty list and keep appending the lines to it and finally use file.writelines (pro: single write operation (?), con: huge list takes up memory)

3. Write each line to the file as it is constructed (pro: no large memory consumed, con: huge number of write operations)

What is the idiomatic/recommended way of writing large number of lines to a file?

Answer 1

Option 3: Write the lines as you generate them.

Writes are already buffered; you don't have to do it manually as in options 1 and 2.

Answer 2

Option #3 is usually the best; normal file objects are buffered, so you won't be performing excessive system calls by writing as you receive data to write.

Alternatively, you can mix option #2 and #3; don't build intermediate lists and call .writelines on them, make the code that would produce said lists a generator function (having it yield values as it goes) or generator expression, and pass that to .writelines . It's functionally equivalent to #3 in most cases, but it pushes the work of iterating the generator to the C layer, removing a lot of Python byte code processing overhead. It's usually meaningless in the context of file I/O costs though.