Python multiprocessing: How to shutdown a long running process that is a sequence of actions

Question

I want to use multiprocessing to run simultaneously several processes of a series of lengthy actions (eg clicking through programs simultaneously using computer vision). I want to implement a stop button in a GUI that would allow me to stop the processes midway - but I'm not sure how I can do something like that.

I know I can use a shutdown event to control the processes if there's a loop - however in my case the target function is not a loop. Here's some code where I basically just did an event check on every line of the target function:

(I deleted a large part of the tkinter code as it's not relevant, the start/stop functions are for their respective buttons in the GUI.)

from multiprocessing import Event, Process
from time import sleep

def start_proc(self):
    self.proc_stop_event.clear()
    for i in range(4):
        p = Process(target=long_sequential_tasks, args=(i,self.proc_stop_event))
        p.start()

def stop_proc(self):
    self.proc_stop_event.set()

#a series of lengthy, different and sequential tasks
def long_sequential_tasks(idx, event):
    if not event.is_set(): <---
        print(str(idx))
        sleep(10) 
    else:
        break
    if not event.is_set(): <---
        print(str(idx * 2))
        sleep(10)
    else:
        break
    if not event.is_set(): <---
        print(str(idx * 3))
        sleep(10)
    else:
        break

Checking the event status every line is obviously not a solution, but I really don't know what else I can do. In the future I want to add queues and locks to synchronize the parallel processes, and apparently process.terminate can leave them hanging.

So if process.terminate is not an option, how can I interrupt a process that's carrying out a sequence of actions in a clean way? Or I can still terminate the processes but do a clean up on the hanging stuff?

Answer 1

Q : how can I interrupt a process that's carrying out a sequence of actions in a clean way?

May implement a signaling/messaging metaplane, using nanomsg or ZeroMQ, that will smart-inform the process ( which is checking for presence of such a <NO-GO> -message, as an example in due places in between the phases of the pure-[SERIAL] workflow check-points ) and once a <NO-GO> -message arrives, the process may perform a graceful-cleanup and smart termination of its own run.

The add-on costs are minimal.

def long_sequential_tasks( idx ):
    #-----------------------------------------------------------------------
    # local
    # mock-up soft-authentcation proxy
    # for otherwise possible ( whitelisting & key-management & other tools available )
    #
    CONTROL_HASH_ID = "asdfhasrtbnastnve9vaergn9ervgnaeragiagiab - WHATEVER"
    LENofMASK = len( CONTROL_HASH_ID )
    #-----------------------------------------------------------------------
    import zmq
    aCtx = zmq.Context()
    aSUB = aCtx.socket( zmq.SUB )
    aSUB.setsockopt(    zmq.SUBSCRIBE, CONTROL_HASH_ID )
    aSUB.setsockopt(    zmq.LINGER, 0 )
    ...
    aSUB.connect( "<protocol>://<address-details>" ) # { ipc:// | tipc:// | norm:// | tcp:// | vmci:// | ...  }

    if (  0 == aSUB.poll( 0, zmq.POLLIN )
       or      aSUB.recv(    zmq.NOBLOCK )[:LENofMASK] != CONTROL_HASH_ID
           ):                    # <--- NO LEGAL EXPLICIT <NO-GO>-MESSAGE HERE
           # continue planned work-step
           ...

    if (  0 == aSUB.poll( 0, zmq.POLLIN )
       or      aSUB.recv(    zmq.NOBLOCK )[:LENofMASK] != CONTROL_HASH_ID
           ):                    # <--- NO LEGAL EXPLICIT <NO-GO>-MESSAGE HERE
           # continue planned work-step
           ...

The requested effect is achieved in a clean way.

Tkinter MVC-actor-based GUI and any other smart-"sources" of control-messages can live with this concept and initiate smart-signaling/messaging outside the world of the tk.mainloop() -scheduler and easily reach inter-platform integrated computing exo-systems ( using this for more than a decade for a remote GUI for a QuantFX prototyping with AI/ML + trading backend ecosystem ).

---

Another way is to implement the exit-plan via a use of a custom-defined signal-handler. This releaves the need to structure the sequence of steps after/before which the exit-plan is being checked in a rather rigid way. Signal handlers can "interrupt" the flow of computations and may (rather brutally) kill its own process (which needs to add some mitigation measures, if graceful terminations are a must), but represents an option, if loooong running workflow-steps are worth stopping upon some external logic, responsible for the exit-plan flagging a <NO-GO> -state.