在这种玩具环境中最好的多处理方法

Question

I'd like to increase the speed of my project using multiprocessing.

from multiprocessing import Queue, Process

def build(something):
    # ... Build something ...
    return something

# Things I want to build.
# Each of these things requires DIFFERENT TIME to be built.
some_things = [a_house, a_rocket, a_car]

#________________________________
# My approach

def do_work(queue, func, args):
    queue.put(func(*args))

# Initialize a result queue
queue = Queue()

# Here I'll need to distribute the tasks (in case there are many)
# through each process. For example process 1 build a house and a rocket 
# and so on. Anyway this is not the case..
procs = [Process(target=do_work, args=thing) for thing in some_things]

# Finally, Retrieve things from the queue
results = []
while not queue.empty():
    results.append(queue.get())

Here the problem is that if a process finish to build its stuff it will wait until other processes will finish while I want such process to do something else.

How can I achieve this? I think I could use a pool of workers but I don't really understand how to use it because I need to retrieve the results. Can someone help with this?

Answer 1

There are a couple of techniques you can use:

1. Use a shared-memory Array to communicate between the main process and all the child processes. Put dicts as input values and set a flag once an output value has been computed.

2. Use Pipes to communicate job init data from the master to the workers, and results back from the workers to the master. This works well if you can serialize the data easily.

Both of these classes are detailed here: http://docs.python.org/2/library/multiprocessing.html