Unable to read stdout from a running process

Question

I have read and tried 8 different methods answered on several questions about this. I opened a process in python and want to read its output even if the process hasn't terminated yet. The process doesn't complete usually for at least 1 minute or until an interrupt is sent. No matter what I try, I can't get it to read the output. I know the command and args I passed work because when I change it to subprocess.call(cmd, args), it prints everything to the screen. I also checked that the process is running with ps -ax. Here is an example of what I'm trying (cat /dev/random is UNRELATED to my project):

proc = subprocess.Popen(["cat", "/dev/random"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Process started.")

Here is what I have tried that has failed so far:

for line in iter(p.stdout.readline, ''):
    strLine = str(line).rstrip()
    print(">>> " + strLine )
    sys.stdout.flush()

And

output, error = proc.communicate()
print output

And

while proc.poll() is None:
    print("Still waiting.")
    print(proc.stdout.readline(1))

There are more solutions I tried that are variations of this but no luck. When the call function is used without changing stdout, everything prints correctly to the console. What am I doing wrong?

I'm using Python 2.6.

Answer 1

I copied your code into a complete function-and-file, adding one change ( repr ) to avoid printing stuff that changes terminal titles and such, giving:

import subprocess
import sys

def tst():
    proc = subprocess.Popen(["cat", "/dev/random"],
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    print("Process started.")
    for line in iter(p.stdout.readline, ''):
        strLine = str(line).rstrip()
        print(">>> " + repr(strLine))
        sys.stdout.flush

tst()

(oops, looks like my cut-and-paste dropped the parentheses on sys.stdout.flush! harmless in this case though)

Running this immediately produces the obvious error:

Process started.
Traceback (most recent call last):
  File "foo.py", line 13, in <module>
    tst()
  File "foo.py", line 8, in tst
    for line in iter(p.stdout.readline, ''):
NameError: global name 'p' is not defined

Fixing that (replacing p with proc ), the example works, for some definition of "works": /dev/random does not stop producing output so it runs forever.

The middle example will be a problem since proc.communicate() is going to read the entire output of the process, which is infinite and hence will run you out of memory (eventually). :-)

The third example works fine.

If you replace the cat /dev/random with something else, you may discover a more interesting and perhaps annoying aspect of Unix/Linux pipelines: a process's stdout stream is normally line buffered if and only if it goes to an "interactive device" (like a terminal window). A pipe is not an "interactive device", so stdout is block-buffered, unless the command in question overrides this itself. This may be the root of the problem I can't reproduce here.

You can work around this by using pseudo-ttys instead of (or in addition to) Python's subprocess module.