Run programs in background and redirect their outputs to file in real time

Question

I want to run several python scripts simultaneously in one bash session and to inspect their outputs respectively in real time. To fulfil this task, I wrote a simple bash script which is shown below:

#!/bin/bash
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

When I use cat 1.output command to check what have been printed in the middle of executing, however, nothing can be seen.

After thinking a while, I realise that the 1.output must be filled when 1.py finishes executing. In other words, the method which I used here is not a real time fashion.

You may propse that to wait for these python scripts' finish. The fact, unfortunately, is that all there python scripts are actually long run programs, they maybe finish after days or months, and that is why I want to inspect their outputs in real time.

Also, you may suggest me to modify the python scripts to print message to file directly instead of stdout . Sorry, the scripts here are too complex to modify, there are chucks of print function in them.

what can I do now?

Answer 1

The -u switch and the equivalent PYTHONUNBUFFERED environment variable forces stdout to be unbuffered. Try this:

#!/bin/bash
python -u 1.py > 1.output &
python -u 2.py > 2.output &
python -u 3.py > 3.output &

or

#!/bin/bash
export PYTHONUNBUFFERED=yes
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

Note that -u has side effects: read the doc to learn more.

Reference:

• https://docs.python.org/2/using/cmdline.html#cmdoption-u
• https://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED

Answer 2

If you can find some sort of main loop in the program, that could be a good indicator of progress, it would be good to do write-to-file. If you say that the output does not fill until the output stream is closed by Python, then this is probably the simplest alternative.

Answer 3

You can try to override stderr/stdout in your scripts by doing:

# in the beginning of your script
import os
import sys

desired_output_file = open('/path/to/file', 'a')
os.dup2(desired_output_file.fileno(), sys.stdout)
os.dup2(desired_output_file.fileno(), sys.stderr)

Then all print calls will write to this special file, not to stdout. However, main problem could be with buffered IO. You have to somehow flush content of such file to the disk.