Python (3.7+) 多处理：用 asyncio 替换 master 和 worker 之间的管道连接以实现 IO 并发

Question

Suppose we have a following toy version of master-worker pipeline to parallel data collection

# pip install gym
import gym
import numpy as np
from multiprocessing import Process, Pipe

def worker(master_conn, worker_conn):
    master_conn.close()

    env = gym.make('Pendulum-v0')
    env.reset()

    while True:
        cmd, data = worker_conn.recv()

        if cmd == 'close':
            worker_conn.close()
            break
        elif cmd == 'step':
            results = env.step(data)
            worker_conn.send(results)

class Master(object):
    def __init__(self):
        self.master_conns, self.worker_conns = zip(*[Pipe() for _ in range(10)])
        self.list_process = [Process(target=worker, args=[master_conn, worker_conn], daemon=True) 
                             for master_conn, worker_conn in zip(self.master_conns, self.worker_conns)]
        [p.start() for p in self.list_process]
        [worker_conn.close() for worker_conn in self.worker_conns]

    def go(self, actions):
        [master_conn.send(['step', action]) for master_conn, action in zip(self.master_conns, actions)]
        results = [master_conn.recv() for master_conn in self.master_conns]

        return results

    def close(self):
        [master_conn.send(['close', None]) for master_conn in self.master_conns]
        [p.join() for p in self.list_process]

master = Master()
l = []
T = 1000
for t in range(T):
    actions = np.random.rand(10, 1)
    results = master.go(actions)
    l.append(len(results))

sum(l)

Because of the Pipe connections between master each worker, for every time step, we have to send a command to the worker through the Pipe, and the worker sends back the results. We need to do this for a long horizon. This will be sometimes a bit slow due to frequent communications.

Therefore, I am wondering if by using latest Python feature asyncio combined with Process to replace Pipe, could it be potentially speedup due to IO concurrency, if I understand its functionality correctly.

Answer 1

Multiprocessing module has already a solution for parallel task processing: multiprocessing.Pool

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(processes=4) as pool:         # start 4 worker processes
        print(pool.map(f, range(10)))       # prints "[0, 1, 4,..., 81]"

You can achieve the same using multiprocessing.Queue . I believe that's how pool.map() is implemented internally.

So, what's the difference between multiprocessing.Queue and multiprocessing.Pipe ? Queue is just a Pipe plus some locking mechanism. Therefore multiple worker processes can share just a single Queue (or rather 2 - one for commands, one for results), but with Pipe each process would need it's own Pipe (or a pair of, or a duplex one), exactly how you are doing it now.

The only disadvantage of Queue is performance - because all processes share one queue mutex it doesn't scale well for many processes. To be sure it can handle tens of thousands items/s I would choose Pipe , but for classic parallel task processing use case I think Queue or just Pool.map() could be OK because they are much easier to use. (Managing processes can be tricky and asyncio doesn't make it easier either.)

Hope that helps, I'm aware that I've answered a bit different question than you've asked :)