Python多处理：检索下一个结果

Question

I'm trying to figure out a good way to use the multiprocessing package in Python 3.6 to run a set of around 100 tasks, with a maximum of 4 of them running simultaneously. I also want to:

1. repeatedly reap the next completed task from the pool and process its return value, until all tasks have either succeeded or failed;
2. make exceptions thrown in any given task non-fatal, so I can still access the results from the other tasks.

I don't need to maintain the order of tasks submitted to the pool (ie I don't need a queue). The total number of tasks ("100" above) isn't prohibitively huge, eg I don't mind submitting them all at once and letting them be queued until workers are available.

I thought that multiprocessing.Pool would be a good fit for this, but I can't seem to find a "get next result" method that I can call iteratively.

Is this something I'm going to have to roll myself from process management primitives? Or can Pool (or another thing I'm missing) support this workflow?

For context, I'm using each worker to invoke a remote process that could take a few minutes, and that has capacity to handle N jobs simultaneously ("4" in my concretized example above).

Answer 1

I came up with the following pattern (shown using 2 workers & 6 jobs, instead of 4 & 100):

import random
import time
from multiprocessing import Pool, TimeoutError
from queue import Queue


def worker(x):
    print("Start: {}".format(x))
    time.sleep(5 * random.random())  # Sleep a random amount of time
    if x == 2:
        raise Exception("Two is bad")
    return x


if __name__ == '__main__':

    with Pool(processes=2) as pool:
        jobs = Queue()
        for i in range(6):
            jobs.put(pool.apply_async(worker, [i]))

        while not jobs.empty():
            j = jobs.get(timeout=1)
            try:
                r = j.get(timeout=0.1)
                print("Done: {}".format(r))
            except TimeoutError as e:
                jobs.put(j)  # Not ready, try again later
            except Exception as e:
                print("Exception: {}".format(e))

Seems to work pretty well:

Start: 0
Start: 1
Start: 2
Done: 1
Start: 3
Exception: Two is bad
Start: 4
Start: 5
Done: 3
Done: 4
Done: 5
Done: 0

I'll see whether I can make a general utility to manage the queueing for me.

The main shortcoming I think it has is that completed jobs can go unnoticed for a while, while uncompleted jobs are polled and possibly time out. Avoiding that would probably require using callbacks - if it becomes a big enough problem, I'll probably add that to my app.