使用python多处理从进程获取结果

Question

I am trying to understand how to use the multiprocessing module in Python. The code below spawns four processes and outputs the results as they become available. It seems to me that there must be a better way for how the results are obtained from the Queue; some method that does not rely on counting how many items the Queue contains but that just returns items as they become available and then gracefully exits once the queue is empty. The docs say that Queue.empty() method is not reliable. Is there a better alternative for how to consume the results from the queue?

import multiprocessing as mp
import time


def multby4_wq(x, queue):
    print "Starting!"
    time.sleep(5.0/x)
    a = x*4
    queue.put(a)


if __name__ == '__main__':
    queue1 = mp.Queue()
    for i in range(1, 5):
        p = mp.Process(target=multbyc_wq, args=(i, queue1))
        p.start()
    for i in range(1, 5): # This is what I am referring to as counting again
        print queue1.get()

Answer 1

Instead of using queue, how about using Pool ?

For example,

import multiprocessing as mp
import time


def multby4_wq(x):
    print "Starting!"
    time.sleep(5.0/x)
    a = x*4
    return a

if __name__ == '__main__':
    pool = mp.Pool(4)
    for result in pool.map(multby4_wq, range(1, 5)):
        print result

Pass multiple arguments

Assume you have a function that accept multiple parameters ( add in this example). Make a wrapper function that pass arguments to add ( add_wrapper ).

import multiprocessing as mp
import time


def add(x, y):
    time.sleep(1)
    return x + y

def add_wrapper(args):
    return add(*args)

if __name__ == '__main__':
    pool = mp.Pool(4)
    for result in pool.map(add_wrapper, [(1,2), (3,4), (5,6), (7,8)]):
        print result