![](/img/trans.png)
[英]How to use Python multiprocessing Pool.map to fill numpy array in a for loop
[英]How to use python multiprocessing Pool.map within loop
我正在使用 Runge-Kutta 运行模拟。 在每个时间步,两个独立变量的两个 FFT 是必要的,可以并行化。 我实现了这样的代码:
from multiprocessing import Pool
import numpy as np
pool = Pool(processes=2) # I like to calculate only 2 FFTs parallel
# in every time step, therefor 2 processes
def Splitter(args):
'''I have to pass 2 arguments'''
return makeSomething(*args):
def makeSomething(a,b):
'''dummy function instead of the one with the FFT'''
return a*b
def RungeK():
# ...
# a lot of code which create the vectors A and B and calculates
# one Kunge-Kutta step for them
# ...
n = 20 # Just something for the example
A = np.arange(50000)
B = np.ones_like(A)
for i in xrange(n): # loop over the time steps
A *= np.mean(B)*B - A
B *= np.sqrt(A)
results = pool.map(Splitter,[(A,3),(B,2)])
A = results[0]
B = results[1]
print np.mean(A) # Some output
print np.max(B)
if __name__== '__main__':
RungeK()
不幸的是,python 在到达循环后会生成无限数量的进程。 之前似乎只有两个进程在运行。 我的记忆也填满了。 添加一个
pool.close()
pool.join()
在循环后面并没有解决我的问题,把它放在循环里面对我来说没有意义。 希望你能帮忙。
将池的创建移到RungeK
函数中;
def RungeK():
# ...
# a lot of code which create the vectors A and B and calculates
# one Kunge-Kutta step for them
# ...
pool = Pool(processes=2)
n = 20 # Just something for the example
A = np.arange(50000)
B = np.ones_like(A)
for i in xrange(n): # loop over the time steps
A *= np.mean(B)*B - A
B *= np.sqrt(A)
results = pool.map(Splitter, [(A, 3), (B, 2)])
A = results[0]
B = results[1]
pool.close()
print np.mean(A) # Some output
print np.max(B)
或者,将其放在主块中。
这可能是多处理工作方式的副作用。 例如,在 MS Windows 上,您需要能够在没有副作用的情况下导入主模块(如创建新进程)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.