subprocess to print stderr and stdout in real-time

Question

Say I have this program printer.py :

#!/usr/bin/env python3

import sys
import time

sys.stdout.write("STDOUT 1\n")
time.sleep(1)
sys.stderr.write("STDERR 2\n")
time.sleep(1)
sys.stdout.write("STDOUT 3\n")
time.sleep(1)
sys.stderr.write("STDERR 4\n")
time.sleep(1)

It prints to stdout and stderr to produce:

./printer.py 
STDOUT 1
STDERR 2
STDOUT 3
STDERR 4

I would like to execute printer.py inside another python script, runner.py , and print in real time both stderr and stdout. The following version of runner.py does not work:

#!/usr/bin/env python3

import sys
import subprocess

def run_command(command):
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
    while True:
        output = process.stdout.readline().decode()
        if output == '' and process.poll() is not None:
            break
        if output:
            print(output.strip())
    rc = process.poll()
    return rc

rc = run_command('./printer.py')

because it prints the stderr lines first in real-time and the stdout lines later all at once:

./runner.py 
STDERR 2
STDERR 4
STDOUT 1
STDOUT 3

How can fix it to have the correct order 1, 2, 3, and 4 in real-time? The closer I could get is by using:

rc = run_command('./printer.py 1>&2')

which is kind of ok, but I wonder whether I could make it do the proper thing and print to stdout and stderr in the same way as printer.py .

---

sys.stdout.flush() as suggested in comments makes no difference:

#!/usr/bin/env python3

import sys
import subprocess

def run_command(command):
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
    while True:
        output = process.stdout.readline().decode()
        if output == '' and process.poll() is not None:
            break
        if output:
            sys.stdout.write(output.strip() + '\n')
            sys.stdout.flush()
    rc = process.poll()
    return rc

rc = run_command('./printer.py')

./runner.py 
STDERR 2
STDERR 4
STDOUT 1
STDOUT 3

The same for print(..., flush=True) . Am I doing something wrong?

Answer 1

I'm collating comments and add a bit of mine. Credit goes to @Barmar and @MarkSetchell.

In the end, I think I'm going for the following solution:

rc = run_command('PYTHONUNBUFFERED=1 ./printer.py')

it should do the same as @MarkSetchell's python -u./printer.py . However, for that I would to explicitly set the path to printer.py and I would rather avoid that. But I'm not sure yet about the pro and cons of each.

unbuffer solution: On my Ubuntu 18 is not installed so I'd rather avoid an additional dependency. As I understand it I would use it as rc = run_command('unbuffer./printer.py') ?

Editing printer.py is not an option for me otherwise adding sys.stdout.flush() after each print or sys.stdout.write should also work.