[英]Asynchronously schedule sequential processes with the multiprocessing.Pool Python class
I'm running simulations using an executable in parallel with the Python multiprocessing.Pool
class as follows: 我正在使用可执行文件与Python
multiprocessing.Pool
类并行运行模拟,如下所示:
self._pool = Pool()
return self._pool.apply_async(run_executable, [],
dict(simulator=self,
params=params,
command=self._command,
results=self._results))
Let's say we want to run 5 simulations on a computer with 4 CPUs, then (if I understand correctly) the processes get scheduled as follows: 假设我们要在具有4个CPU的计算机上运行5个仿真,然后(如果我理解正确的话)按以下方式安排进程:
The problem arises when I need the results of a previous simulation to start a new simulation with adjusted parameters. 当我需要先前模拟的结果来启动具有调整参数的新模拟时,就会出现问题。 If we use the same figure as before and let simulation
i.j+1
follow on simulation ij
, the scheduling should look as follows for the easy case where we have a number of simulations equal to the number of CPUs: 如果我们使用与之前相同的数字,并让模拟
i.j+1
跟随模拟ij
,那么对于模拟次数等于CPU数量的简单情况,调度应如下所示:
How can I wait for the right process to finish, do the post-processing and then start a new simulation again in asynchronous mode? 如何等待正确的过程完成,进行后处理,然后以异步模式再次开始新的仿真? (I don't really care if this gets done by the same CPU, I just want to use the maximum amount of CPU power available).
(我不太在乎是否由同一个CPU完成,我只想使用可用的最大CPU功率)。
Replace your code with this and have a look what it does. 以此替换您的代码,看看它的作用。
self._pool = Pool()
def process_ended_callback(result):
print(self._command, "resulted in", result)
return self._pool.apply_async(run_executable, [],
dict(simulator=self,
params=params,
command=self._command,
results=self._results),
process_ended_callback)
It should print which command just ended. 它应该打印哪个命令刚刚结束。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.