当 stderr 被重定向到 subprocess.STDOUT 时 subprocess.stdout.readline() 挂起

Question

I've been looking at other similar questions, but I don't think anyone has covered this particular case.

I've found that using stderr=subprocess.STDOUT in my Popen call causes stdout.readline() to hang forever in rare cases; the best example I have is an attempt at an internal SSH command that sets up a tunnel:

ssh -N -f -L 81:<SOME URL>:80 <SOME OTHER URL>

When I try to run this command in Popen with stderr redirected to stdout, it prints the 2 outputs and then hangs on stdout.readline() forever:

import subprocess as sp
cmd = "ssh -N -f -L 81:<SOME URL>:80 <SOME OTHER URL>"

p = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.STDOUT, shell=True, env=os.environ, encoding="utf-8")
while True:
    out = p.stdout.readline()
    if out:
        print(out)
    if p.poll() is not None:
        break

I'm redirecting stderr to stdout so that I can have both streams in one place.

Is there a way I can check if there's a line to read BEFORE trying to read it with readline()?

Also, why doesn't readline() simply return "" if the process is finished? This seems like a huge miss.

Answer 1

To anyone finding this, I worked around this with a (not my own) clever use of signal .

All this does is set up a straight timeout so that when your code runs, you're only giving it N seconds before it has to complete, or you're killing it.

This Timeout code has been immensely useful in many places for me; you can import signal and then straight copy/paste this in to your work:

# This stuff is so when we get SIGALRM from the timeout functionality we can handle it instead of
# crashing to the ground
class TimeOutError(Exception):
    pass
def raise_timeout(var1, var2):
    raise TimeOutError
signal.signal(signal.SIGALRM, raise_timeout)

Once you have this set, you're now free to set an alarm with signal.alarm(<SOME SECONDS>) and do except TimeOutError in your code; which is what I did for this problem:

import subprocess as sp
import signal

class TimeOutError(Exception):
    pass
def raise_timeout(var1, var2):
    raise TimeOutError
signal.signal(signal.SIGALRM, raise_timeout)

cmd = "ssh -N -f -L 81:<SOME URL>:80 <SOME OTHER URL>"

p = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.STDOUT, shell=True, env=os.environ, encoding="utf-8")

while True:
    out = ""
    signal.alarm(1)
    try:
        out = p.stdout.readline()
    except TimeOutError as e:
        # Is the process complete?
        if p.poll() is not None:
            break
        else:
            pass

    # Turn the alarm back off
    signal.alarm(0)

    if out:
        print(out)
    if p.poll() is not None:
        break