[英]Working withmultiprocessing and h5py
I get an error when trying to run a command with joblib/multiprocessing in parallel: 尝试使用joblib / multiprocessing并行运行命令时出错:
Here the traceback: 追溯:
Process PoolWorker-263:
Traceback (most recent call last):
File "/home/marcel/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/marcel/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/marcel/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/home/marcel/.local/lib/python2.7/site-packages/joblib/pool.py", line 363, in get
File "_objects.pyx", line 240, in h5py._objects.ObjectID.__cinit__ (h5py/_objects.c:2994)
TypeError: __cinit__() takes exactly 1 positional argument (0 given)
As you can see from the error message I work with data loaded using h5py. 正如您从错误消息中看到的,我使用h5py加载的数据。 To complicate things further the routine I want to parallelize uses numba in one of its subroutines, but I hope that does not matter. 为了使事情复杂化,我想要并行化的例程在其子程序之一中使用numba,但我希望这无关紧要。
Here is a running example, which you can copy and paste: 这是一个运行示例,您可以复制和粘贴:
from joblib import Parallel,delayed
import numpy as np
import h5py as h5
import os
def testfunc(h5data, row):
# some very boneheaded CPU work
data_slice = h5data[:,row,...]
ma = np.mean(data_slice, axis = 1)
x = row
return ma, x
def run():
data = np.random.random((100,100,100))
print data
f_out = h5.File('tmp.h5', 'w')
dset = f_out.create_dataset('mydata', data = data )
f_out.close()
f_in = h5.File('tmp.h5', 'r')
h5data = f_in['mydata']
pool = Parallel(n_jobs=-1,verbose=1,pre_dispatch='all')
results = pool(delayed(testfunc)(h5data, i) for i in range(h5data.shape[1]))
f_in.close()
os.remove('tmp.h5')
if __name__ == '__main__':
run()
Any ideas, what I'm doing wrong? 任何想法,我做错了什么?
Edit: Okay at least I can exclude numba from the list of evildoers... 编辑:好吧,至少我可以从恶人列表中排除numba ......
1You can try to replace ˋjoblib with [pathos][1] which replaces
pickle with
dill`. 1你可以尝试with [pathos][1] which replaces
pathos with [pathos][1] which replaces
joblib with [pathos][1] which replaces
with
莳萝with [pathos][1] which replaces
pickle。 This solves generally all pickling issues. 这通常解决了所有酸洗问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.