Python: Reading a subprocess' stdout without printing to a file

Question

I have an executable function called BOB.exe which prints an insane amount of text to stdout with only short pauses. BOB also has a habit of freezing so I wrote a python monitor function which, using the subprocess module, calls the BOB executable, directs it to a temporary file and watches the temp file's size to see if it has crashed. This is my current solution:

#!/usr/bin/python
from subprocess import Popen
import tempfile, time
def runBOB(argsList):

    # Create a temporary file where BOB stdout will be piped
    BOBout = tempfile.NamedTemporaryFile()
    BOBoutSize = 0

    # Start the subprocess of BOB
    BOBsp = Popen(argsList, stdout=BOBout)

    while True:
        # See if subprocess has finished
        if BOBsp.poll() is not None:
            BOBout.close() # Destroy the temp file
            return 0

        # if the size of the stdout file has increased, BOB.exe is still running
        BOBoutSizeNew = os.path.getsize(BOBout.name)
        if BOBoutSizeNew > BOBoutSize:
            BOBoutSize = BOBoutSizeNew
        else: # if not, kill it
            BOBsp.kill()
            BOBout.close() # Destroy the temp file
            return 1

        # Check every 10 seconds
        time.sleep(10)

However, this is incredibly slow and I think the writing to file is the cause. Is there a more efficient way to do this such as watching the stdout stream and then immediately sending it to Null? Anything that cuts down on the megabytes of printing being done would probably help. Is there another way of seeing if the exe has crashed? I should probably note that I don't care about the stdout, it is all going to be ignored anyway

Thank you for your help!

Answer 1

You can use stdout=subprocess.PIPE to tell subprocess to give you ability to read the output of the subprocess without storing it to a file. The tricky part is to do so asynchronously, so as not to deadlock when BOB.exe freezes. An easy way to do so is with a helper thread; although Python has a bad reputation with threads, this is actually a good use case for threading where GIL doesn't get in the way.

Simply create a helper thread that does nothing except reading the output from the file handle corresponding to Bob's output. The helper thread immediately discards the output and increments a byte counter. The main thread implements exactly the same logic you had previously, but using the in-memory counter instead of re-examining the file size. When Bob finishes or is killed by the main thread, the helper thread will receive EOF and exit.

Here is an untested implementation of the above:

#!/usr/bin/python
import subprocess
import threading
import time
import os

bytes_read = 0

def readBOB(pipe):
    global bytes_read
    bytes_read = 0
    while True:
        # Wait for some data to arrive. This must use os.read rather
        # than pipe.read(1024) because file.read would block us if less
        # than 1024 bytes of data arrives. (Reading one byte at a time
        # with pipe.read(1) would work, but would be too slow at
        # consuming large amounts of data.)
        s = os.read(pipe.fileno(), 1024)
        if not s:
            return  # EOF
        # we are the only writer, so GIL serves as the lock
        bytes_read += len(s)

def runBOB(argsList):
    # Start the subprocess of BOB
    BOBsp = subprocess.Popen(argsList, stdout=subprocess.PIPE)

    thr = threading.Thread(target=readBOB, args=(BOBsp.stdout,))
    thr.start()
    old_bytes_read = -1

    while True:
        # See if subprocess has finished
        if BOBsp.poll() is not None:
            return 0

        # if the size of the stdout has increased, BOB.exe is still running
        new_bytes_read = bytes_read

        if new_bytes_read > old_bytes_read:
            old_bytes_read = new_bytes_read
        else: # if not, kill it (readBOB will exit automatically)
            BOBsp.kill()
            return 1

        # Check every 10 seconds
        time.sleep(10)