reading file while it is being written

Question

Ok. This is weird. I started with an empty file called foo.txt and in a python3 interpreter, wrote the following code:

>>> import time
>>> f = open('foo.txt', 'r')
>>> while True:
...     print(f.readlines())
...     f.seek(0)
...     time.sleep(5)
... 

and I started seeing output like this:

[]
0
[]
0

So far, so good. Then I opened another window, and ran the following:

echo "line number 1" >> foo.txt

and my output changed to:

['line number 1\n']
0

Now in the other window, I opened file via vim and added one new line such that final state of the file looked like this (I closed vim after that)

line number 1
line number 2

Strangely enough, I was still seeing the "previous" output in the interpreter

['line number 1\n']
0
['line number 1\n']
0

I expected to see two lines in the interpreter now, but that was not the case. For some reason, writing file via echo was behaving differently from writing file via vim .

At that point, I decided to see whether the file descriptor in python is still pointing to the original file.

>>> print(f.name)
foo.txt

Confused, I decided to check lsof output

$ lsof | grep foo.txt
python3.6 .... foo.txt~

Ah ha! The tilde at the end meant that the open fd is actually now pointing to something else. Running the same experiment on ubuntu (the above was on MacOS), lsof had an interesting output

$ lsof | grep foo.txt
python3.6 .... temp.txt~ (deleted)

So, when the file was opened in vim , operating system (?) at that point decided to create a copy of the file for the already opened fd. (But didn't do this when I edited the file via echo )

Does anyone know why this happens only for vim, and not for echo?

Answer 1

When you save a file (eg, foo.txt ) in vim, the editor will typically follow these steps to avoid losing content if something goes wrong while writing to the file:

1. Open a new file with a temporary name in the same directory. (When I tried this, it used a four-digit number as the filename; by way of example, let's call it 1234 .)

2. Write the contents of your buffer to the temporary file 1234 .

3. Check what the permissions and extended attributes were on foo.txt .

4. Delete foo.txt~ , if it exists.

5. Rename foo.txt to foo.txt~ .

6. Rename 1234 to foo.txt , and set its permissions and extended attributes to match what it got earlier.

7. Delete foo.txt~ . (If you :set backup , this step is skipped.)

This means that, when you write to a file in vim, the original file gets replaced with a completely new file. The file handle you had open in Python follows the original (now deleted) file.

Appending to a file in the shell using >> does not follow any of these steps. It just writes extra data to the end of the existing file, like you'd expect.