python modify mutable iterator

Question

The code is like following:

f=open('test.txt')
file=iter(f)

When I do

next(file)

It will print line by line of file. But when I modified the test.txt file and saved it, the next(file) still printed the original file content.

Does iterator store the complete file in the memory? If not why the content of the file didn't get updated?

Answer 1

No, as an iterator the file object stores only a look-ahead buffer, not the complete file, in memory. This makes it efficient for large files.

Since there is this look-ahead buffer, changes made to the file won't be reflected to the next method. However, you can use the seek method to clear this buffer so the next call to the next method will return the updated content:

f.seek(f.tell()) # seek the current position only to clear the look-ahead buffer
print(next(f)) # prints the updated next line from the current position

Answer 2

Let's suppose open() reads 2 letters at a time. (Actual value is io.DEFAULT_BUFFER_SIZE )

f=open('test.txt')

You've created a file object, _io.TextIOWrapper which, oversimplified, is like [{read from 0 to io.DEFAULT_BUFFER_SIZE of test.txt}, ...}

file=iter(f)

You've created a iterator of _io.TextIOWrapper with data like this: [{read from 0 to 1}, ... {read from n-1 to n}]

next(file)

next() has went through the first item of file , read it, and printed it.

Let's learn from an example.

Reading normally

test.txt

what a beautiful day

We will open the file, iter(), and list() to open and iter through all of it and make a list.

In [1]: f = open('test.txt')

In [2]: list(iter(f))
Out[2]: ['what a beautiful day']

Just as expected.

File changes after open()

In [1]: f = open('test.txt')

We've opened the file.

We will now append hello open() to test.txt.

test.txt

what a beautiful day

hello open()

and then iter() and list() it.

In [2]: list(iter(f))
Out[2]: ['what a beautiful day\n', '\n', 'hello open()']

The changed contents are seen. We can see that open() does not actually read the file.

File changes after iter()

In [1]: f = open('test.txt')

In [2]: i = iter(f)

We've opened the file and iter() d.

We will now append hello iter()

test.txt

what a beautiful day

hello open()

hello iter()

and then list() it.

In [3]: list(i)
Out[3]: ['what a beautiful day\n', '\n', 'hello open()\n', '\n', 'hello iter()']

The changed contents are seen. We can also see that iter() does not actually read the file.