[英]How to pass multiple arguments in Multiprocessing executor.map() function
I have been watching several videos on Multiprocessing map
function.我一直在观看有关多处理
map
function 的几个视频。
I know that I can send one list as an argument to the function I want to target with Multiprocessing, and that will call the same function n times (dependent upon the size of that passed list).我知道我可以将一个列表作为参数发送到我想要使用多处理的 function,这将调用相同的 function n 次(取决于传递的列表的大小)。
What I am struggling to do is what if I want to pass multiple arguments to that function?我正在努力做的是,如果我想将多个 arguments 传递给那个 function 怎么办?
I basically have a List whose size is n
(it can vary but for current case, its 209)我基本上有一个大小为
n
的列表(它可以变化,但对于当前情况,它是 209)
My function requires 3 arguments...我的 function 需要 3 arguments...
I could have used the 2nd and 3rd arguments as global variables, but that doesn't work for me because I have to call the map function in a while loop... and in every another iteration, the values of these two will change.我本可以使用第二个和第三个 arguments 作为全局变量,但这对我不起作用,因为我必须在 while 循环中调用 map function ...并且在每一次迭代中,这两个的值都会改变。
My function returns two values which I need to access in the function from where it was called.我的 function返回两个值,我需要从调用它的地方访问 function。 This is what I have tried but it didn't work for me,
这是我尝试过的方法,但对我不起作用,
def main_fun():
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(MyFun, (row, pop[0].data, fitness) for row in range(0, len(pop[0].data)))
for result in results:
print(result)
I also tried to use ZIP
function but again, with no success.我也尝试使用
ZIP
function 但再次没有成功。
If your second and third arguments to your worker function (ie the first argument to map
), then you can use method functools.partial
to have the second and third arguments specified without resorting to the use of global variables.如果您的第二个和第三个 arguments 给您的工作人员 function(即第一个参数
map
),那么您可以使用方法functools.partial
指定第二个和第三个 arguments,而无需使用全局变量。 If your worker functions is, for example, foo
, then:例如,如果您的辅助函数是
foo
,那么:
from concurrent.futures import ProcessPoolExecutor
from functools import partial
def foo(idx: int, lst: list, int_value: int):
...
def main():
with ProcessPoolExecutor() as executor:
worker = partial(foo, lst=pop[0].data, int_value=fitness)
executor.map(worker, range(0, len(pop[0].data)))
if __name__ == '__main__':
main()
So now we only have to pass to map
function worker
, which will be called two fixed arguments, and a single iterable argument.所以现在我们只需要传递给
map
function worker
,这将被称为两个固定的 arguments 和一个可迭代的参数。
If you are executing the map
call in a loop, you will, of course, create a new worker
functions by passing to functools.partial
new arguments.如果您在循环中执行
map
调用,您当然会通过传递给functools.partial
new arguments 创建一个新的worker
函数。
For example:例如:
from concurrent.futures import ProcessPoolExecutor
from functools import partial
def foo(idx: int, lst: list, int_value: int):
print(idx, lst[idx] * int_value, flush=True)
def main():
l = [3, 5, 7]
fitness = 9
with ProcessPoolExecutor() as executor:
worker = partial(foo, lst=l, int_value=fitness)
executor.map(worker, range(0, len(l)))
if __name__ == '__main__':
main()
Prints:印刷:
0 27
1 45
2 63
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.