[英]Python multiprocessing using pickle, kwargs, and function references
I have a situation where I encounter problems using any efficent multiprocessing framework in python (no matter whether scoop or multiprocessing). 我遇到的情况是在python中使用任何有效的多处理框架都遇到问题(无论是独家还是多处理)。
I have the following situation: 我有以下情况:
Here is the example: 这是示例:
import multiprocessing as mp
class Foo:
def __init__(self, f):
self.f = f
class Bar:
def __init__(self, foo, **kwargs):
self.args = kwargs
self.foo = foo
def execute(self):
pool = mp.Pool(5)
f = lambda x : self.foo.f(**x)
args = [self.args] * 10
results = pool.map(f, args)
if __name__ == '__main__':
def anything(**kwargs):
print(kwargs['z'])
return kwargs['x'] * kwargs['y']
foo = Foo(anything)
args = {'x':10, 'y':27, 'z':'Hello'}
bar = Bar(**args)
I know that functions must be on module level in order to be pickable. 我知道功能必须在模块级别才能被选择。 Is there any way to be able to get the function pickable?
有什么方法可以使函数可选取吗? Unfortunately, I am not very experiences in Python OOP, so probably I am missing an important point!
不幸的是,我对Python OOP的经验不是很丰富,所以可能我错过了重要的一点! Thank you!
谢谢!
EDIT: Unfortunately, even with using the module "multiprocess" which uses dill instead of pickle (thanks to Mike McKerns) it is not guaranteed that my problem is solved. 编辑:不幸的是,即使使用使用莳萝代替泡菜的模块“ multiprocess”(感谢Mike McKerns),也不能保证我的问题得到了解决。 For some short runs of my program, things are fine.
对于我的程序的一些短期运行,一切都很好。 For some reasons, multiprocess seems to generate race conditions as I get following error:
由于某些原因,当我遇到以下错误时,多进程似乎会生成竞争条件:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/multiprocess/pool.py", line 389, in _handle_results
task = get()
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 209, in loads
return load(file)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 199, in load
obj = pik.load()
File "/usr/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1096, in load_global
klass = self.find_class(module, name)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 353, in find_class
return StockUnpickler.find_class(self, module, name)
File "/usr/lib/python2.7/pickle.py", line 1132, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'Individual'
(Individual is a class which is used by my program [genetic algorithm using deap]) Any idea? (个人是我的程序[使用deap的遗传算法]使用的类)。
(repeating the comments above) (重复以上评论)
Substitute the multiprocess
package for multiprocessing
and the code should work with no other changes. 替代
multiprocess
包multiprocessing
和代码应该没有其他的变化工作。 This is because multiprocess
is a fork of multiprocessing
that uses dill
instead of pickle
… so you are able to serialize almost anything in python, including stuff you write in the interpreter session. 这是因为
multiprocess
是一个叉multiprocessing
使用dill
,而不是pickle
...所以你能够在序列几乎蟒任何东西,包括你的解释器会话写的东西。 That's the only change made for the fork of multiprocessing
. 这是对
multiprocessing
的分支所做的唯一更改。
See https://stackoverflow.com/a/21345273/2379433 and https://stackoverflow.com/a/21345308/2379433 and https://stackoverflow.com/a/21345423/2379433 and etc. 参见https://stackoverflow.com/a/21345273/2379433和https://stackoverflow.com/a/21345308/2379433和https://stackoverflow.com/a/21345423/2379433等。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.