Python subprocess call with output and timeout

Question

Summary : I want to start an external process from Python (version 3.6), poll the result nonblocking, and kill after a timeout.

Details : there is an external process with 2 "bad habits":

1. It prints out the relevant result after an undefined time.
2. It does not stop after it printed out the result.

Example : maybe the following simple application resembles mostly the actual program to be called ( mytest.py ; source code not available):

import random
import time

print('begin')
time.sleep(10*random.random())
print('result=5')
while True: pass

This is how I am trying to call it:

import subprocess, time
myprocess = subprocess.Popen(['python', 'mytest.py'], stdout=subprocess.PIPE)
for i in range(15):
    time.sleep(1)
    # check if something is printed, but do not wait to be printed anything
    # check if the result is there
    # if the result is there, then break
myprocess.kill()

I want to implement the logic in comment.

Analysis

The following are not appropriate:

• Use myprocess.communicate() , as it waits for termination, and the subprocess does not terminate.
• Kill the process and then call myprocess.communicate() , because we don't know when exactly the result is printed out
• Use process.stdout.readline() because that is a blocikg statement, so it waits until something is printed. But here at the end does not print anything.

The type of the myprocess.stdout is io.BufferedReader . So the question practically is: is there a way to check if something is printed to the io.BufferedReader , and if so, read it, but otherwise do not wait?

Answer 1

I think I got the exact package you need. Meet command_runner , which is a subprocess wrapper and allows:

• Live stdout / stderr output
• timeouts regardless of execution
• process tree including child processes killing in case of timeout
• stdout / stderr redirection to queues, files or callback functions

Install with pip install command_runner

Usage:

from command_runner import command_runner

def callback(stdout_output):
    # Do whatever you want here with the output
    print(stdout_output)

exit_code, output = command_runner("python mytest.py", timeout=300, stdout=callback, method='poller')

if exit_code == -254:
    print("Oh no, we got a timeout")
    print(output)

# Check for good exit_code and full stdout output here

If timeout is reached, you'll get exit_code -254 but still get to have output filled with whatever your subprocess wrote to stdout/stderr.

Disclaimer: I'm the author of command_runner

Additional non blocking examples using queues can be seen on the github page.