[英]Cannot pass argument to method using multiprocessing.Pool
My program takes several arguments where one of them is called challenges
which receives integer value from the command line. 我的程序有几个参数,其中一个称为
challenges
,它从命令行接收整数值。 I want to use multiprocessing
by passing the value of challenges
to a self-defined method generation
: 我想通过将
challenges
的价值传递给自定义方法generation
来使用multiprocessing
:
import multiprocessing
gen = generator.GenParentClass()
mlp = multiprocessing.Pool(processes=multiprocessing.cpu_count())
X, y = mlp.imap_unordered(gen.generation, [args.challenges])
The method generation
in class GenParentClass
has this simple signature: GenParentClass
类中的方法generation
具有以下简单签名:
def generation(self, num):
#some stuff
However, I get this error: 但是,我收到此错误:
Traceback (most recent call last):
File "experiments.py", line 194, in <module>
X, y = mlp.imap_unordered(gen.generation, [args.challenges])
File "/anaconda/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
I don't know how to solve this problem. 我不知道如何解决这个问题。 Everything seems to me correct!!
在我看来一切都正确! Any help os appreciated.
任何帮助操作系统表示赞赏。
The multiprocess
module serializes ( pickles
) the arguments to imap_unordered
. multiprocess
imap_unordered
模块将参数序列化( pickles
)到imap_unordered
。 It appears that the function gen.generation
is an instance method (defined within a class), which means it cannot be pickled, hence your error. 看来函数
gen.generation
是实例方法(在类中定义),这意味着它不能被腌制,因此会出错。
Edit: Here is a possible workaround that defines the function outside of the class, and adds additional argument(s) to that function, which are filled in using partial
from itertools
: 编辑:这是一个可能的解决方法,它在类外部定义函数,并向该函数添加其他参数,这些参数使用
itertools
中的partial
来填充:
import multiprocessing
from functools import partial
class GenParentClass(object):
a = None
b = None
def __init__(self, a, b):
self.a = a
self.b = b
# define this outside of GenParentClass (i.e., top level function)
def generation(num, x, y):
return x+y+num
gen = GenParentClass(3, 5)
mlp = multiprocessing.Pool(processes=multiprocessing.cpu_count())
R = mlp.imap_unordered(partial(generation, x=gen.a, y=gen.b), [1,2,3])
print([r for r in R]) # prints "[9, 10, 11]"
More information on pickle-ability is available here . 有关腌制能力的更多信息,请参见此处 。
More information on functools is available here . 有关functools的更多信息,请参见此处 。
Edit 2: If you use multiprocess.Pool
and the function definition uses qualified variable names self.a
and self.b
, you can do this without rewriting the function outside the class, but you won't be able to retrieve the output, and the state of gen2 will not change (defeating the purpose of calling the function at all). 编辑2:如果您使用
multiprocess.Pool
,并且函数定义使用合格的变量名称self.a
和self.b
,则可以执行此操作而无需在类外部重写函数,但是您将无法检索输出,并且gen2的状态不会改变(根本无法实现调用该函数的目的)。
gen2 = GenParentClass(4, 6)
p = {}
for key in range(5):
p[key] = multiprocessing.Process(target = GenParentClass.altgen, args = (gen2, key,))
p[key].start()
for key in p:
p[key].join()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.