tail and less commands not monitoring file in real time

Question

I'm looking for a way to monitor a file that is written to by a program on Linux. I found the tail -F command in here , and also recommended was less +FG . I tested it by running tail -F file in one terminal, and a simple python script:

import time

for i in range(20):
  print i
  time.sleep(0.5)

in another. I redirected the output to the file:

python script.py >> file

I expected that tail would track the file contents and update the display in fixed intervals, instead it only shows what was written to the file after the command terminates.

The same thing happens with less +FG and also if I watch the output from cat . I've also tried using the usual redirect which truncates the file > instead of >> . Here it says the file was truncated, but still does not track it in real time.

Any idea why this doesn't work? (It's suggested here that it might be due to buffered writes, but since my script runs over 10 seconds, I suspect this might not be the cause)

Edit: In case it matters, I'm running Linux Mint 18.1

Answer 1

Python's standard out is buffered. If when you close the script / script is done, you see all the output - that's definitely buffer issue.

You can use this instead:

import time
import sys

for i in range(20):
    sys.stdout.write('%d\n' % i)
    sys.stdout.flush()
    time.sleep(0.5)

I've tested it and it prints values in real time. To overcome buffer issue, after each .write() method I use .flush() force "flushing" the buffer.

---

Additional options from the comments:

• Use the original print statement with sys.stdout.flush() afterwords
• Run the python script with python -u for unbuffered binary stdout and stderr

Answer 2

Regarding jon1467 answer (sorry can't comment your answer), your understanding of redirection is wrong.

Try this :

dd if=/dev/urandom > test.txt

while looking at the file size with :

ls -l test.txt

You'll see the file grow while dd is running.

Vinny's answer is correct, python standard output is buffered. The more common way to the "buffering effect" you notice is by flushing the stdout as Vinny showed you.

You could also use -u option to disable buffering for the whole python process, or you could just reopen standard output with a buffer size of 0 as below (in python2 at least):

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)