Python: catch exception from subprocess while iterating through stdout

Question

I am trying to get an exception from a subprocess. I can get it if I use .communicate , but I would like to avoid using that since I am streaming output from the subprocess as it occurs, and dont want to wait until the whole subprocess is complete. Also assume the entire subprocess can take a very long time. Was wondering how I can catch an exception thrown while streaming the stdout from subprocess.

Consider the example below, so I would like to get version #1 working, version #2 kinda works, but dont want it that way.

In main.py

import subprocess


class ExtProcess():
    def __init__(self, *args):
        self.proc = subprocess.Popen(['python', *args], stdout=subprocess.PIPE)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if self.proc.returncode:
                raise Exception("error here")
            if not line:
                raise StopIteration
            return line


def run():
    ## version #1
    reader = ExtProcess("sample_extproc.py")
    for d in reader:
        print(f"got: {d}")

    ## version #2
    # proc = subprocess.Popen(['python', "sample_extproc.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # output, error = proc.communicate()
    # print("got:", output)
    # if proc.returncode:
    #     raise Exception(error)

def main():
    try:
        print("start...")
        run()
        print("complete...")
    except Exception as e:
        print(f"Package midstream error here: {str(e)}")
    finally:
        print("clean up...")


if __name__ == "__main__":
    main()

In sample_extproc.py

for x in range(10):
    print(x)
    if x == 3:
        raise RuntimeError("catch me")

I would like to get an output of something like below, from version #1:

start...
got: b'0\r\n'
got: b'1\r\n'
got: b'2\r\n'
got: b'3\r\n'
Package midstream error here: b'Traceback (most recent call last):\r\n  File "sample_extproc.py", line 4, in <module>\r\n    raise RuntimeError("catch me")\r\nRuntimeError: catch me\r\n'
clean up...

Basically it iterates through the stdout from the subprocess, then prints the exception when it occurs, then continues performing cleanup.

Answer 1

Here's the answer to my question below, really based on @CharlesDuffy's comment:

In short, make sure to have stderr=subprocess.PIPE in ExtProcess class, then the answer is in version #3, where after iterating through the stdout, we utilize .wait() and returncode to check if there was an error, if so raise an exception, grabbing the error from stderr.read() to be catched in the parent/main.

import subprocess

class ExtProcess():
    def __init__(self, *args):
        self.proc = subprocess.Popen(['python', *args], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if not line:
                raise StopIteration
            return line


def run():
    ## version #1
    # reader = ExtProcess("sample_extproc.py")
    # for d in reader:
    #     print(f"got: {d}")

    ## version #2
    # proc = subprocess.Popen(['python', "sample_extproc.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # output, error = proc.communicate()
    # print("got:", output)
    # if proc.returncode:
    #     raise Exception(error)

    ## version #3
    reader = ExtProcess("sample_extproc.py")
    for d in reader:
        print(f"got: {d}")
    reader.proc.wait()
    if reader.proc.returncode:
       raise Exception(reader.proc.stderr.read())

def main():
    try:
        print("start...")
        run()
        print("complete...")
    except Exception as e:
        print(f"Package midstream error here: {str(e)}")
    finally:
        print("clean up...")


if __name__ == "__main__":
    main()