通过管道传递大型数组时，Python多处理卡住

Question

I'm using multiprocessing in python and try to pass a large numpy array to a subprocess though a pipe. It works well with a small array but hangs for larger arrays without returning an error.

I believe that the pipe is blocked and already read a bit about it but cannot figure out how to solve the problem.

def f2(conn, x):
    conn.start()
    data = conn.recv()
    conn.join()

    print(data)
    do_something(x)

    conn.close()

if __name__ == '__main__':
    data_input = read_data()    # large numpy array
    parent_conn, child_conn = Pipe()

    p = multiprocessing.Pool(processes=8)      
    func = partial(f2, child_conn)

    parent_conn.send(data_input)
    parent_conn.close()

    result = p.map(func, processes)

    p.close()
    p.join()

Answer 1

Ignoring all the other problems in this code (you don't have an x to pass to map , you don't use the x f2 receives, mixing Pool.map with Pipe is usually the wrong thing to do), your ultimate problem is the blocking send call being performed before a worker process is available to read from it.

Assuming you really want to mix map with Pipe , the solution is to launch the map asynchronously before beginning the send , so there is something on the other side to read from the Pipe while the parent is trying to write to it:

if __name__ == '__main__':
    data_input = read_data()    # large numpy array
    parent_conn, child_conn = Pipe()

    # Use with to avoid needing to explicitly close/join
    with multiprocessing.Pool(processes=8) as p:
        func = partial(f2, child_conn)

        # Launch async map to ensure workers are running
        future = p.map_async(func, x)

        # Can perform blocking send as workers will consume as you send
        parent_conn.send(data_input)
        parent_conn.close()

        # Now you can wait on the map to complete
        result = future.get()

As noted, this code will not run due to the issues with x , and even if it did, the Pipe documentation explicitly warns that two different processes should not be reading from the Pipe at the same time.

If you wanted to process the data in bulk in a single worker, you'd just use Process and Pipe , something like:

def f2(conn):
    data = conn.recv()
    conn.close()
    print(data)

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()

    proc = multiprocessing.Process(target=f2, args=(child_conn,))
    proc.start()

    data_input = read_data()    # large numpy array
    parent_conn.send(data_input)
    parent_conn.close()

    proc.join()

If you wanted to process each element separately across many workers, you'd just use Pool and map :

def f2(x):
    print(x)

if __name__ == '__main__':
    data_input = read_data()    # large numpy array
    with multiprocessing.Pool(processes=8) as p:   
        result = p.map(f2, data_input)