[英]multiprocessing full capacity in Python
I wrote the following code which call function (compute_cluster) 6 times in parallel (each run of this function is independent of the other run and each run write the results in a separate file), the following is my code: 我编写了以下代码,该代码并行调用6次函数(compute_cluster)(此函数的每次运行独立于其他运行,每次运行将结果写入单独的文件中),以下是我的代码:
global L
for L in range(6,24):
pool = Pool(6)
pool.map(compute_cluster,range(1,3))
pool.close()
if __name__ == "__main__":
main(sys.argv)
despite the fact that I'm running this code on a I7 processor machine, and no matter how much I set the Pool to it's always running only two processes in parallel so is there any suggestion on how can I run 6 processes in parallel? 尽管我在I7处理器计算机上运行此代码,并且无论我将Pool设置为多少,它始终只能并行运行两个进程,所以有关于如何并行运行6个进程的建议吗? such that the first three processes use L=6 and call compute_cluster with parameter values from 1:3 in parallel and at the same time the other three processes run the same function with the same parameter values but this time the Global L value is 7 ?
这样前三个进程使用L = 6并并行调用参数值为1:3的compute_cluster,同时其他三个进程使用相同的参数值运行相同的函数,但是这次的全局L值为7? any suggestions is highly appreciated
任何建议都受到高度赞赏
There are a few things wrong here. 这里有些错误。 First, as to why you always only have 2 processes going at a time -- The reason is because
range(1, 3)
only returns 2 values. 首先,关于为什么每次总是只有2个进程的原因-原因是因为
range(1, 3)
只返回2个值。 So you're only giving the pool 2 tasks to do before you close it. 因此,在关闭池之前,您只给池2个任务。
The second issue is that you're relying on global state. 第二个问题是您要依赖全局状态。 In this case, the code probably works , but it's limiting your performance since it is the factor which is preventing you from using all your cores.
在这种情况下,代码可能会起作用 ,但是它会限制您的性能,因为这是阻止您使用所有内核的因素。 I would parallelize the L loop rather than the "inner" range loop.
我将并行化L循环,而不是“内部”范围循环。 Something like 1 :
类似于1的东西:
def wrapper(tup):
l, r = tup
# Even better would be to get rid of `L` and pass it to compute_cluster
global L
L = l
compute_cluster(r)
for r in range(1, 3):
p = Pool(6)
p.map(wrapper, [(l, r) for l in range(6, 24)])
p.close()
This works with the global L because each spawned process picks up its own copy of L
-- It doesn't get shared between processes. 这适用于全局L,因为每个产生的进程都选择自己的
L
副本-进程之间不会共享。
1 Untested code 1个 未经测试的代码
As pointed out in the comments, we can even pull the Pool
out of the loop: 正如评论中指出的,我们甚至可以将
Pool
退出循环:
p = Pool(6)
p.map(wrapper, [(l, r) for l in range(6, 24) for r in range(1, 3)])
p.close()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.