Run subprocess in python and get stdout and kill process on timeout

Question

Here is my code, it launches a subprocess, waits till it ends and returns stdout, or a timeout happens and it raises exception. Common use is print(Run('python --version').execute())

class Run(object):
    def __init__(self, cmd, timeout=2*60*60):
        self.cmd = cmd.split()
        self.timeout = timeout
        self._stdout = b''
        self.dt = 10
        self.p = None

    def execute(self):
        print("Execute command: {}".format(' '.join(self.cmd)))

        def target():
            self.p = Popen(self.cmd, stdout=PIPE, stderr=STDOUT)
            self._stdout = self.p.communicate()[0]

        thread = Thread(target=target)
        thread.start()

        t = 0
        while t < self.timeout:
            thread.join(self.dt)
            if thread.is_alive():
                t += self.dt
                print("Running for: {} seconds".format(t))
            else:
                ret_code = self.p.poll()
                if ret_code:
                    raise AssertionError("{} failed.\nretcode={}\nstdout:\n{}".format(
                        self.cmd, ret_code, self._stdout))
                return self._stdout

        else:
            print('Timeout {} reached, kill task, pid={}'.format(self.timeout, self.p.pid))
            self.p.terminate()
            thread.join()
            raise AssertionError("Timeout")

The problem is following case. The process that I launch spawns more child processes. So when the timeout is reached, I kill main process (the one I srarted using my class) with self.p.terminate() , the children are remaining and my code hangs on line self._stdout = self.p.communicate()[0] . And execution continues if I manually kill all child processes.

I tried soulution when instead of self.p.terminate() I kill whole process tree.

This also does not work if the main process finished by itself and its children are existing on their own, and I have no ability to find and kill them. But they are blocking self.p.communicate() .

Is there way to effectively solve this?

Answer 1

You could use the ProcessWrapper from the PySys framework - it offers alot of this functionality as an abstraction in a cross platform way ie

import sys, os
from pysys.constants import *
from pysys.process.helper import ProcessWrapper
from pysys.exceptions import ProcessTimeout

command=sys.executable
arguments=['--version']
try:
    process = ProcessWrapper(command, arguments=arguments, environs=os.environ, workingDir=os.getcwd(), stdout='stdout.log', stderr='stderr.log', state=FOREGROUND, timeout=5.0)
    process.start()
except ProcessTimeout:
    print "Process timeout"
    process.stop()

It's at SourceForge ( http://sourceforge.net/projects/pysys/files/ and http://pysys.sourceforge.net/ ) if of interest.