打印子进程的标准输出 pipe 忽略最后几行

Question

I'm trying to run a sub processes and watching his stdout until I find desirable string. this is my code:

    def waitForAppOutput(proc, word):
        for stdout_line in iter(proc.stdout.readline, b''):
           print stdout_line
           if word in stdout_line.rstrip():
              return;

    p = Popen(["./app.sh"], shell=True, stdin=PIPE ,stdout=PIPE, stderr=PIPE)
    waitForAppOutput(p,"done!")

the issue here is that for some reason the function waitForAppOutput stop printing stdout few lines before the "done." which is the last line that should appear in the stdout . I assume iter(proc.stdout.readline, b'') is blocking and readline is not able to read the last lines of the stdout.

any idea what is the issue here?

Answer 1

You have a misspelling: it should be waitForAppOutput instead of waitForAppOutout . How does this even run at all? And when you are invoking a command using a shell, you should not be passing an array of strings but rather one single string.

Normally one should use the communicate method on the return subprocess object from the Popen call to prevent potential deadlocks (which seems to be what you are experiencing). This returns a tuple: (stdout, stderr) , the stdout and stderr output strings:

from subprocess import Popen, PIPE


def waitForAppOutput(stdout_lines, word):
    for stdout_line in stdout_lines:
       print stdout_line
       if word in stdout_line.rstrip():
          return;

p = Popen("./app.sh", shell=True, stdout=PIPE, stderr=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = "command line 1\ncommand line 2\n"
stdout, stderr = p.communicate(expected_input)
stdout_lines = stdout.splitlines()
waitForAppOutput(stdout_lines, "done!")

The only issue is if the output strings are large (whatever your definition of large might be), for it might be memory-inefficient or even prohibitive to read the entire output into memory. If this is your situation, then I would try to avoid the deadlock by piping only stdout .

from subprocess import Popen, PIPE


def waitForAppOutput(proc, word):
    for stdout_line in iter(proc.stdout.readline, ''):
       print stdout_line
       if word in stdout_line.rstrip():
          return;

p = Popen("./app.sh", shell=True, stdout=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = "command line 1\ncommand line 2\n"
p.stdin.write(expected_input)
p.stdin.close()
waitForAppOutput(p, "done!")
for stdout_line in iter(p.stdout.readline, ''):
    pass # read rest of output
p.wait() # wait for termination

Update

Here is an example using both techniques that runs the Windows sort command to sort a bunch of input lines. This works particularly well both ways because the sort command does not start output until all the input has been read, so it's a very simple protocol in which deadlocking is easy to avoid. Try running this with USE_COMMUNICATE set alternately to True and False :

from subprocess import Popen, PIPE

USE_COMMUNICATE = False

p = Popen("sort", shell=True, stdout=PIPE, stdin=PIPE, universal_newlines=True)
expected_input = """q
w
e
r
t
y
u
i
o
p
"""
if USE_COMMUNICATE:
    stdout_lines, stderr_lines = p.communicate(expected_input)
    output = stdout_lines
else:
    p.stdin.write(expected_input)
    p.stdin.close()
    output = iter(p.stdout.readline, '')
for stdout_line in output:
    print stdout_line,
p.wait() # wait for termination

Prints:

e
i
o
p
q
r
t
u
w
y