subprocess.Popen handling stdout and stderr as they come

Question

I'm trying to process both stdout and stderr from a subprocess.Popen call that captures both via subprocess.PIPE but would like to handle the output (for example printing them on the terminal) as it comes.

All the current solutions that I've seen will wait for the completion of the Popen call to ensure that all of the stdout and stderr is captured so that then it can be processed.

This is an example Python script with mixed output that I can't seem to replicate the order when processing it in real time (or as real time as I can):

$ cat mix_out.py

import sys

sys.stdout.write('this is an stdout line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')

The one approach that seems that it might work would be using threads, because then the reading would be asynchronous, and could be processed as subprocess is yielding the output.

The current implementation of this just process stdout first and stderr last, which can be deceiving if the output was originally alternating between both:

cmd = ['python', 'mix_out.py']

process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=True,
    **kw
)

if process.stdout:
    while True:
        out = process.stdout.readline()
        if out == '' and process.poll() is not None:
            break
        if out != '':
            print 'stdout: %s' % out
            sys.stdout.flush()

if process.stderr:
    while True:
        err = process.stderr.readline()
        if err == '' and process.poll() is not None:
            break
        if err != '':
            print 'stderr: %s' % err
            sys.stderr.flush()

If I run the above (saved as out.py ) to handle the mix_out.py example script from above, the streams are (as expected) handled in order:

$ python out.py
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line

I understand that some system calls might buffer, and I am OK with that, the one thing I am looking to solve is respecting the order of the streams as they happened.

Is there a way to be able to process both stdout and stderr as it comes from subprocess without having to use threads? (the code gets executed in restricted remote systems where threading is not possible).

The need to differentiate stdout from stderr is a must (as shown in the example output)

Ideally, no extra libraries would be best (eg I know pexpect solves this)

A lot of examples out there mention the use of select but I have failed to come up with something that would preserve the order of the output with it.

Answer 1

Sorry if I misunderstand the question...but if you are looking for a way of having subprocess.Popen output to stdout/stderr in realtime , you should be able to achieve that with:

import sys, subprocess
p = subprocess.Popen(cmdline,
                     stdout=sys.stdout,
                     stderr=sys.stderr)

Probably, stderr=subprocess.STDOUT , may simplify for your filtering?

If that is not what you are/were looking for, sorry. But hopefully it will fit others needs.

Answer 2

I found working example here (see listing of capture_together.py ). Compiled C++ code that mixes cerr and cout executed as subprocess on both Windows and UNIX OSes. Results are identitical

Answer 3

I was able to solve this by using select.select()

process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=True,
    **kw
)

while True:
    reads, _, _ = select(
        [process.stdout.fileno(), process.stderr.fileno()],
        [], []
    )

    for descriptor in reads:
        if descriptor == process.stdout.fileno():
            read = process.stdout.readline()
            if read:
                print 'stdout: %s' % read

        if descriptor == process.stderr.fileno():
            read = process.stderr.readline()
            if read:
                print 'stderr: %s' % read
        sys.stdout.flush()

    if process.poll() is not None:
        break

By passing in the file descriptors to select() on the reads argument (first argument for select() ) and looping over them (as long as process.poll() indicated that the process was still alive).

No need for threads. Code was adapted from this stackoverflow answer