Pass Lock object to class that inherits multiprocessing.Process

Question

How can I pass a Lock object to subclass of multiprocessing.Process ? I've tried this and I faced pickling error.

from multiprocessing import Process
from threading import Lock

class myProcess (Process):

    def setLock (self , lock) :
        self.lock = lock

    def run(self) :
        with self.lock :
            # do stuff

if __name__ == '__main__' :
    lock = Lock()
    proc1 = myProcess()
    proc1.setLock(lock)

    proc2 = myProcess()
    proc2.setLock(lock)

    proc1.start()
    proc2.start()

There are many answered questions about passing lock to multiprocessing.Pool but none of them solved my problem with OOP approach usage of Process . If I wanna make a global lock , where should I define it and where can I pass it to myProcess objects ?

Answer 1

You can't use a threading.Lock for multiprocessing, you need to use a multiprocessing.Lock .

You get the pickling-error because a threading.Lock can't be pickled and you are on a OS which uses "spawn" as default for starting new processes (Windows or macOS with Python 3.8+).

Note that on a forking OS (Linux, BSD...), with using threading.Lock , you wouldn't get a pickling error, but the lock would be silently replicated, not providing the synchronization between processes you intended.

Using a separate function for setting the lock is possible, but I would prefer passing it as argument to Process.__init__() along with possible other arguments.

import time
from multiprocessing import Process, Lock, current_process


class MyProcess(Process):

    def __init__(self, lock, name=None, args=(), kwargs={}, daemon=None):
        super().__init__(
            group=None, name=name, args=args, kwargs=kwargs, daemon=daemon
        )
        # `args` and `kwargs` are stored as `self._args` and `self._kwargs`
        self.lock = lock

    def run(self) :
        with self.lock :
            for i in range(3):
                print(current_process().name, *self._args)
                time.sleep(1)


if __name__ == '__main__' :

    lock = Lock()
    p1 = MyProcess(lock=lock, args=("hello",))
    p2 = MyProcess(lock=lock, args=("world",))

    p1.start()
    p2.start()

    p1.join()  # don't forget joining to prevent parent from exiting too soon.
    p2.join()

Output:

MyProcess-1 hello
MyProcess-1 hello
MyProcess-1 hello
MyProcess-2 world
MyProcess-2 world
MyProcess-2 world

Process finished with exit code 0