在 multiprocessing.Pool.apply_async function 中使用 multiprocessing.Pipe() 时发生死锁

Question

I try to use multiprocessing.Pipe() as communication tool in multi processes. But when I pass pipe into the pool.apply_async() as a parameter, there is a deadlock problem.Why?

The code and output are:

# coding=utf-8
from multiprocess import pool
from multiprocessing import Pipe, Pool, set_start_method, get_context, Queue, Manager, Process
import time


def worker_process(name, _out_pipe, _in_pipe):

    # _out_pipe.close()

    for x in range(10):
        _in_pipe.send(name + ':' + str(x))
        print(name + ' send value :' + str(x))
        time.sleep(0.1)

    # _in_pipe.close()

if __name__ == '__main__':
    set_start_method('spawn')
    print(get_context())
    # with pool.Pool() as pool:
    with Pool() as pool:
        pool.apply_async(worker_process, ('son_p1', out_pipe, in_pipe))
        pool.apply_async(worker_process, ('son_p2', out_pipe, in_pipe))
        pool.apply_async(worker_process, ('son_p3', out_pipe, in_pipe))
        # pool.apply(worker_process, ('son_p1', out_pipe, in_pipe))
        # pool.apply(worker_process, ('son_p2', out_pipe, in_pipe))
        # pool.apply(worker_process, ('son_p3', out_pipe, in_pipe))
        pool.close()
        pool.join()

    while out_pipe.poll():
        print(out_pipe.recv())

    # in_pipe.close()
    # out_pipe.close()

Process ForkPoolWorker-2:
Process ForkPoolWorker-5:
Process ForkPoolWorker-1:
Process ForkPoolWorker-6:
Process ForkPoolWorker-8:
Process ForkPoolWorker-9:
Process ForkPoolWorker-7:
Process ForkPoolWorker-4:
Traceback (most recent call last):
  File "/Users/zhaolong/PycharmProjects/pipEnvGrpc/pipe_example.py", line 34, in <module>
    pool.join()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 662, in join
    self._worker_handler.join()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):

But when I replace multiprocessing.Pool() to multiprocess.pool.Pool() or multiprocessing.Pool().apply_async() to multiprocessing.Pool().apply() , the program will run normally.Why?

Answer 1

Try using spawn to create new processes instead of fork as per this blog post .

from multiprocessing import set_start_method

if __name__ == '__main__':
    set_start_method("spawn")
    print(get_context())

Additionally, try to terminate your pool using finally. This has appeared as an issue elsewhere .

try:
    with Pool() as pool:
        pool.apply_async(worker_process, ('son_p1', out_pipe, in_pipe))
        pool.apply_async(worker_process, ('son_p2', out_pipe, in_pipe))
        pool.apply_async(worker_process, ('son_p3', out_pipe, in_pipe))
        pool.close()
        pool.join()

    while out_pipe.poll():
        print(out_pipe.recv())
finally:
    pool.terminate()