Multiprocessing, Python3, Windows: TypeError: can't pickle _thread.lock objects

Question

On a Windows 10 system running Python 3.6, when trying to use multiprocessing.Process to create a new rq worker,

multiprocessing.Process(target=Worker(qs).work, kwargs={'burst': True}).start()     

we encounter the error

TypeError: can't pickle _thread.lock objects

and

OSError: [WinError 87] The parameter is incorrect

rq.SimpleWorker was used instead of rq.Worker because Windows does not support the fork function used by rq.Worker .

Question: What is causing this error? How can we solve it?

from rq import SimpleWorker, Connection

def needMoreWorkers():
    return True

if __name__ == '__main__':
    qs = sys.argv[1:] or ['default']
    with Connection(connection=my_redis_conn):
        if needMoreWorkers():
            multiprocessing.Process(target=SimpleWorker(qs).work, kwargs={'burst': True}).start()     

Error Traceback

Traceback (most recent call last):
File "WorkerFactory.py", line 53, in <module>
    main(qs)
File "WorkerFactory.py", line 45, in main
    multiprocessing.Process(target=SimpleWorker(qs)).start()
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

(test) C:\Code\test\source>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
File "C:\Users\x\AppData\Local\Continuum\anaconda3\envs\test\lib\multiprocessing\reduction.py", line 82, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect   

Answer 1

From what I can see, the Pickle module is causing the issue. it must be imported somewhere in another module. anyway, pickle is used for saving data, but it apparently can't "pickle" a thread.lock object. As far as how to fix it, check all of your function calls and make sure you're passing in the correct variable types.

Answer 2

It's possible that _thread.lock is actually a method instead of a regular class object. pickle didn't pickle methods, last I checked. However, dill does. You might try using dill instead of pickle and see if it works.

If you're on Ubuntu, you can install it with sudo apt-get install python3-dill .

Python allows for functional programming, which means that methods and functions can be passed as arguments to functions.

To find out exactly what _thread.lock is, you can use the help() function on it.