简体繁体 English

加入多处理池中的已完成进程

[英]Join finished processes in multiprocessing pool

原文 2020-11-30 23:00:51 9 1 python/ multiprocessing/ pool

I am using a multiprocessing.Pool of n worker processes to map some function over a fully-defined iterable of size m (ie the length and all the elements are known in advance).我正在使用n工作进程的multiprocessing.Pool到 map 一些 function 在完全定义的大小为m的迭代上（即长度和所有元素都是预先知道的）。

Is there a possibility to join the workers as the pool is still executing, provided that there less remaining tasks than active workers?如果剩余任务少于活跃的工作人员，是否有可能在池仍在执行时加入工作人员？ Or, more generally, how can we achieve such behaviour (ie without necessarily a Pool)?或者，更一般地说，我们如何才能实现这样的行为（即不一定是池）？

For instance, given n processes and m tasks, with m >= n , there will be a point in the execution when n == m , so, from this point onward, each process finishing a task will be joined.例如，给定n进程和m个任务，其中m >= n ，当n == m时，执行中会有一个点，因此，从这一点开始，每个完成任务的进程都将被加入。

Thanks!谢谢！

1 个解决方案

Were you trying to reduce memory leakage or high memory usage in the idle workers?您是否试图减少闲置工人的 memory 泄漏或高 memory 使用？

Pool can take maxtasksperchild which restarts worker process once the worker has finished this number of tasks. Pool可以使用maxtasksperchild ，一旦 worker 完成了这个数量的任务，它就会重新启动 worker 进程。 This was originally used for avoiding memory usage piling up during the execution of a worker process.这最初用于避免在工作进程执行期间堆积 memory 使用量。 When set to 1, this may kill the idle worker, but I'm not sure.当设置为 1 时，这可能会杀死空闲的工作人员，但我不确定。 Depending on your real use case, this could be helpful even if it does not kill idle worker.根据您的实际用例，即使它不会杀死空闲的工作人员，这也可能会有所帮助。