[英]Combine output multiprocessing python
This question is related to a previous question I asked , and it seems like a simple question, but I have a hard time finding useful information or tutorials about the topic of multiprocessing. 这个问题与我之前提出的问题有关,这似乎是一个简单的问题,但我很难找到有关多处理主题的有用信息或教程。
My problem is that I would like to combine the produced data into one big array and then store it in my hdf file. 我的问题是我想将生成的数据合并到一个大数组中,然后将其存储在我的hdf文件中。
def Simulation(i, output):
# make a simulation which outputs it resutlts in A. with shape 4000,3
A = np.array([4000,3])
output.put(A)
def handle_output(output):
hdf = pt.openFile('simulation.h5',mode='w')
hdf.createGroup('/','data')
# Here the output should be joined somehow.
# I would like to get it in the shape [4000,3,10]
output.get()
hdf.createArray('/data','array',A)
hdf.close()
if __name__ == '__main__':
output = mp.Queue()
jobs = []
proc = mp.Process(target=handle_output, args=(output, ))
proc.start()
for i in range(10):
p = mp.Process(target=Simulation, args=(i, output))
jobs.append(p)
p.start()
for p in jobs:
p.join()
output.put(None)
proc.join()
What you really need is a multiprocessing Pool 你真正需要的是一个多处理池
Just do something like: 做一些像:
def Simulation(i):
return output
p = mp.Pool(16)
result = p.map(Simulation,range(10))
result = np.array(result).reshape(...)
p.close()
p.join()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.