Python多处理队列内存管理

Question

Say I have the Main processes and 2 additional processes A and B . In this program A is supposed to send data to B. If we have some code like this:

from multiprocessing import Process, Queue

def process_a(iterable, q):

    for x in iterable:
        q.put(x)

def process_b(q):

    while some_condition():
        x = q.get()


iterable = some_iterable()
q = Queue()

pa = Process(target=process_a, args=(iterable, q))
pb = Process(target=process_b, args=(q,))

pa.start()
pb.start()

pa.join()
pb.join()

given that the Queue q was created in the Main process, does the data flow like this?

A => Main => B

If so, is there a way to have a Queue initialized on B and passed to A such that data goes directly from A to B skipping Main ?

Answer 1

given that the Queue q was created in the Main process, does the data flow like this?

A => Main => B

No. As explained in the docs , a Queue is just an auto-synchronizing wrapper around a Pipe . When you pass a Queue to a child, you're just passing that Pipe and some locks.

And the Pipe is just a wrapper around an operating system pipe. When you pass a Pipe to a child, you're just passing the pipe's file descriptor/handle.

Ignoring the locks, process A is basically just writing to a pipe, and process B is just reading from it.

The locks do make things a bit more complicated (and may also mean that process A spins up a hidden background thread), but they still don't involve the main process at all.

Unless the main process calls a method on a queue, it has nothing to do with that queue at all.