[英]Multiprocessing a loop in Python
I have the following code that I want to speed up (using multiprocessing). 我有以下代码,我想加快(使用多处理)。
def main(arg1):
data=[]
#Calculate new argument arg2
for i in range(n):
data.append(function(z,i,arg2))
Where z
in a 2D array. 其中
z
在2D数组中。 My idea was to do it the following way, but I am not sure this will speed up the process. 我的想法是按照以下方式进行,但我不确定这会加快这个过程。
from multiprocessing import Pool
import itertools
def function_star(a_b_c):
return function(*a_b_c)
def main(arg1):
#Calculate new argument arg2
pool=Pool()
i=range(n)
out=pool.map(function_star, i, itertools.repeat(z),itertools.repeat(arg2) )
pool.close()
if __name__=="__main__":
main(arg1)
Is this indeed the most efficient way to speed up the process? 这确实是加速这一进程的最有效方式吗?
If I interpret your code block correctly you want to have function
called with always the same z
, and arg1
but with i
being a range (I am a bit unsure because the code you pasted will not work, as map
only takes one iterable and you're giving 3) 如果我正确地解释你的代码块,你想让
function
调用始终是相同的z
和arg1
但是i
是一个范围(我有点不确定,因为你粘贴的代码不起作用,因为map
只需要一个iterable而你'给予3)
If this is the case, then partial
solves your issue: 如果是这种情况,那么
partial
解决您的问题:
from multiprocessing import Pool
from functools import partial
def function(i, z, arg2):
print(z, i, arg2)
def main(arg1):
#Calculate new argument arg2
pool=Pool()
i=range(n)
out=pool.map(partial(function, z=5, arg2=3), i)
pool.close()
if __name__=="__main__":
main(arg1)
note that you need to change the order of arguments in your function so that the changing i
parameter is at first position. 请注意,您需要更改函数中参数的顺序,以便更改
i
参数位于第一个位置。
If you care about speed, you should add a third argument to map
with the chunksize. 如果你关心速度,你应该添加第三个参数来
map
chunksize。 This makes that a process asks for a chunksize
packet from the main process so you have a smaller numer of communications between the main process and the child processes. 这使得进程要求来自主进程的
chunksize
数据包,因此主进程和子进程之间的通信数量较少。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.