Safe way to terminate a python subprocess?

Question

I'm using python's multiprocessing module to handle a number of functions concurrently. Each spawned-process' function gets some initial input arguments, and a Pipe connection to send its results back. For various reasons, I must use individual processes like this, ie tools like Pool.map_async() -methods are off the table.

Occasionally, I need to terminate a process which takes too long to finish.

According to the Process documentation :

Warning: If this method is used when the associated process is using a pipe or queue then the pipe or queue is liable to become corrupted and may become unusable by other process. Similarly, if the process has acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock.

I'm not worried about the first part, as each process gets their own pipe object, but how do I determine if a process has 'acquired a lock or semaphore' , and/or terminate in a way that's safe for the remainder of my program?

Answer 1

Q : " Safe way to terminate a python subprocess ?"

Well, if there were some, you would never run into this.

If your actual needs can justify the costs of doing so right and well, the best workaround would be to learn from Masters of reliable processing, designing robust, resilient, self-healing systems - like Ms.Margaret HAMILTON (MIT) pioneered in NASA operated Apollo Moon Landing Programme, designing the AGC ( Apollo Guidance Computer ) so right and so well, it could survive its own deadlocking risks, preventing the Eagle lander from crashing the Moon surface.

Best inspirations come from distributed-computing , available for pythonistas if designing safe and self-healing autonomous components using a robust and independent many-node-to-many-node communications-plane frameworks ZeroMQ or nanomsg.

Answer 2

As a side note: It might be worthwhile to check why your subprocesses are taking 'too long to finish'.

As for the warning, it relates to when you 'lock' resources for use. For example:

# function to withdraw from account 
def withdraw(balance, lock):     
    for _ in range(10000): 
        lock.acquire() 
        balance.value = balance.value - 1
        lock.release() 

source: https://www.geeksforgeeks.org/synchronization-pooling-processes-python/

If you would terminate the subprocess after it has performed lock.acquire() and before it performed lock.release() , you would have a deadlock situation.

So the question is, do you use any threading.Lock or threading.Semaphore objects in the processes that you want to terminate?

I hope this helps in understanding whether it is safe to terminate your subprocess/thread.

EDIT: By the way, you should also consider using kill() instead of terminate().

Answer 3

On *nix, you can try sending SIGINT to the process instead of terminating/killing it and catch KeyboardInterrupt exception for the cleanup:

from multiprocessing import Process
import os
import time
import signal

def f():
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        print('Do the emergency cleanup here')

p = Process(target=f)
p.start()
os.kill(p.pid, signal.SIGINT)
p.join()