在附加模式下我的文件是否在RAM中打开？

Question

I have written a code which keep on append the file. Here is the code for it:

writel = open('able.csv','a',encoding='utf-8',errors='ignore')
with open('test','r',encoding='utf-8',errors='ignore') as file:
    for i in file.readlines():
        data = functionforprocess(i)
        if data is not "":
            writel.write(data)
        if count% 10000 == 0:
            log = open('log','w')
            log.write(str(count))
            log.close()

My question is: whether the file that I have opened in the append mode is available in RAM? Does that file is acting like a buffer, means If I store the data in variable and then write the variable to file is equal to open a file in append mode and write directly?

Kindly, get me out of this confusion.

Answer 1

Appending is a basic function of file I/O and is carried out by the operating system. For instance, fopen with mode a or a+ is part of the POSIX standard. With file I/O, the OS will also tend to buffer reads and writes; for instance, for most purposes it's not necessary to make sure that the data that you've passed to write is actually on the disk all the time. Sometimes it sits in a buffer somewhere in the OS; sometimes the OS dumps these buffers out to disk. You can force writes using fsync if it's important to you; this is also a really good reason to make sure that you always call close on your open file objects when you're done with them (or use a context manager); if you forget, you might get weird behaviour because of those buffers hanging around in the OS.

So, to answer your question. The file that you opened is most likely in RAM at any given moment. However, as far as I know, it's not available to you. You can interact with the data in the file using file I/O methods, but it's not like there's a buffer that you can get the memory address of, and read back what you just wrote. As to if append-mode writing is equivalent to storing something in a buffer and then writing to disk, I guess I would say no. Any kind of file I/O write will probably be buffered the same way by the OS, and the reason this is efficient is that the OS gets to make the decision on when to flush the buffers. If you store things in a variable and then write them out atomically to disk, you get to decide when the writes take place.

Answer 2

The signature of the open function is:

open(file, mode=’r’, buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

If you open in "a" (append) mode, it means: open for writing, appending to the end of the file if it exists. There is nothing about buffering.

Buffering can be customized with the buffering parameter. Quoting the doc:

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. When no buffering argument is given, the default buffering policy works as follows:

• Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device's “block size” and falling back on io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will typically be 4096 or 8192 bytes long.
• “Interactive” text files (files for which isatty() returns True) use line buffering. Other text files use the policy described above for binary files.

In your example, your file is opened for append in text mode.

So, only a chunk of your data is stored in RAM during writing. If you write a "big" data, it will be divided into several chunks.