在Python中使用多处理模块时，如何确保每个进程使用大致相同的时间？

Question

Currently I am working on an asynchronous gradient algorithm with Python multiprocessing module, the main idea is that I run multiple processes that update an array of global parameters asynchronously. I have finished most of the framework but I got a problem that some processes seems to "get stuck" sometimes while other are still running, that causes this algorithm less effective. So I am wondering if there are good ways to make sure that they use roughly the same amount of time?

Thanks!

Answer 1

This depends almost entirely on the problem you try to tackle. If you distribute a large task to several workers and one unpredictably gets a much larger chunk than the others, you will have this situation.

There are several options to avoid it:

1. Try to estimate the effort for each chunk more precisely. Depending on your task, this might be possible. The chunks with the most predicted effort should be split.
2. A very common way to approach this is to split the task into lots of very small chunks, many more than workers are present. Then feed all chunks into a queue and let your workers eat their chunks from the queue. This way when a worker receives an easy chunk it will finish it fast and take at once the next chunk from the queue, thus not ending up idle while other workers seem to be "stuck" with their harder chunk.

A real deadlock will not be fixed of course by whatever approach.