Kill children of Python subprocess after subprocess.TimeoutExpired is thrown

Question

I am calling a shell script fom within Python, that spawns multiple child processes. I want to terminate that process and all of its children, if it did not finish after two minutes.

Is there any way I can do that with subprocess.run or do I have to go back to using Popen? Since run is blocking, I am not able to save the pid somewhere to kill the children in an extra command. A short code example:

try:
    subprocess.run(["my_shell_script"], stderr=subprocess.STDOUT, timeout=120)
except subprocess.TimeoutExpired:                                                                      
    print("Timeout during execution")

Answer 1

This problem was reported as a bug to the Python developers. It seems to happen specifically when stderr or stdout is redirected. Here is a more correct version of @Tanu's code.

import subprocess as sp

try:
    proc = sp.Popen(['ls', '-l'], stdout=sp.PIPE, stderr=sp.PIPE)
    outs, errs = proc.communicate(timeout=120)
except sp.TimeoutExpired:
    proc.terminate()

Popen doesn't accept timeout as a parameter. It must be passed to communicate . On Posix OSs, terminate is more gentle than kill , in that it reduces the risk of creating zombie processes.

Answer 2

Quoting from the docs:

subprocess.run - This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.

Don't have to use Popen() if you don't want to. The other functions in the module, such as .call(), .Popen().

There are three 'file' streams: stdin for input, and stdout and stderr for output. The application decides what to write where; usually error and diagnostic information to stderr, the rest to stdout. To capture the output for either of these outputs, specify the subprocess.PIPE argument so that the 'stream' is redirected into your program.

To kill the child process after timeout:

import os
import signal
import subprocess

try:
    proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE, timeout=120)
except subprocess.TimeoutExpired:
    os.kill(proc.pid, signal.SIGTERM)