How to create a multiprocessing Queue with set_start_method?

Question

I am trying to create multiprocessing.Queue that is referenced in a Pool created within main() . However, using this code:

import multiprocessing as mp

def f(_):
    print(b)

def main():
    print("main")
    print(b)

    print("running f")
    f(0)

    with mp.Pool(2) as pool:
        pool.map(f, list(range(2)))

if __name__ == '__main__':
    mp.set_start_method('spawn')
    b=mp.Queue()

    main()

I get NameError: name 'b' is not defined . This code runs fine without the mp.set_start_method('spawn') call. How can I define a global Queue object that will be visible to f() and still use the mp.set_start_method('spawn') ?

Answer 1

Use the initializer and initargs arguments of multiprocessing.pool.Pool to initialize global variable b for each pool process:

import multiprocessing as mp

# Initialize each pool process:
def init_pool(q):
    global b

    b = q

def f(_):
    print(b, b.get())

def main():
    print("main")
    print(b)

    print("running f")
    f(0)

    with mp.Pool(2, initializer=init_pool, initargs=(b,)) as pool:
        pool.map(f, list(range(2)))

if __name__ == '__main__':
    mp.set_start_method('spawn')
    b=mp.Queue()
    b.put(1)
    b.put(2)
    b.put(3)

    main()

Prints:

main
<multiprocessing.queues.Queue object at 0x00000192041E38E0>
running f
<multiprocessing.queues.Queue object at 0x00000192041E38E0> 1
<multiprocessing.queues.Queue object at 0x000001843E12A190> 2
<multiprocessing.queues.Queue object at 0x000001843E12A190> 3