Kill python script that executes multiple subprocesses

Question

I have a python script that executes other python scripts using subprocess . My problem is that if and I want to termintate the full process, with for ex. crt-c, it kills only the current iteration but starts the execution of the following one. The main script:

for fold in range(10):
    comand = "python train_kfolds_pos.py {} {}".format(current_model,fold)
    print(comand)
    process = subprocess.Popen(comand, shell=True, stdout=subprocess.PIPE)
    process.wait()
    text = process.communicate()[0].decode("utf-8")
    print(text)

Answer 1

since you're not checking the return code of wait() the program continues with the loop, even if the sub-process is interrupted.

You should check it (and use a list of args/remove shell=True )

for fold in range(10):
    comand = ["python","train_kfolds_pos.py",current_model,str(fold)]
    print(comand)
    process = subprocess.Popen(comand, stdout=subprocess.PIPE)
    rc = process.wait()
    # quit loop if return code is not 0
    if rc:
       print("aborted, returncode {}".format(rc))
       break
    text = process.communicate()[0].decode("utf-8")
    print(text)

a side effect of this is that if the program terminates abnormally for some other reason, it stops too.

Aside: you could think about not calling a python subprocess but directly call a function within the python train_kfolds_pos.py script and parameters. Exceptions would propagate naturally (so would keyboard interrupt exceptions). Something like:

import train_kfolds_pos
for fold in range(10):
    train_kfolds_pos.compute(current_model,fold)