[英]How to append multiprocessed result that is in a loop?
I want to utilize multiple processors to calculate a function for two lists of values.我想利用多个处理器来计算两个值列表的函数。 In the test case below, my idea is that I have two lists: c = [1,2,3], and c_shift = [2,3,4].在下面的测试用例中,我的想法是我有两个列表:c = [1,2,3] 和 c_shift = [2,3,4]。 I want to evaluate a function for a single value in each list, and append two separate solution arrays.我想为每个列表中的单个值计算一个函数,并附加两个单独的解决方案数组。
import numpy as np
import multiprocessing as mp
def function(x,a,b,c):
return a*x**2+b*x+c
def calculate(x,a,b,c):
c_shift = c+1
result = []
result_shift = []
for i in range(len(c)):
process0 = mp.Process(target = function, args = (x,a,b,c[i]))
process1 = mp.Process(target = function, args = (x,a,b,c_shift[i]))
process0.start()
process1.start()
process0.join()
process1.join()
# After it finishes, how do I append each list?
return np.array(result), np.array(result_shift)
if __name__ == '__main__':
x = np.linspace(-1,1,50)
a = 1
b = 1
c = np.array([1,2,3])
calculate(x,a,b,c)
When each process finishes, and goes through join()
, how do I append process0
to result = []
and process1
to result_shift = []
?当每个进程完成并通过join()
,如何将process0
附加到result = []
并将process1
到result_shift = []
?
The structure of the returned results should have the form:返回结果的结构应具有以下形式:
result = [ [1 x 50], [1 x 50], [1 x 50] ]结果 = [ [1 x 50], [1 x 50], [1 x 50] ]
result_shifted = [ [1 x 50], [1 x 50], [1 x 50] ] result_shifted = [ [1 x 50], [1 x 50], [1 x 50] ]
Slightly different approach but I think this is what you were looking to do?稍微不同的方法,但我认为这就是你想要做的?
import multiprocessing
import numpy as np
from functools import partial
def your_func(c, your_x, a, b):
results = []
for c_value in c:
results.append(a * your_x ** 2 + b * your_x + c_value)
return results
def get_results(c_values):
your_x = np.linspace(-1, 1, 50)
a = 1
b = 1
with multiprocessing.Pool() as pool:
single_arg_function = partial(your_func, your_x=your_x, a=a, b=b)
out = pool.map(single_arg_function, c_values)
return out
if __name__ == "__main__":
c_values = [np.array([1, 2, 3]), np.array([1, 2, 3]) + 1]
out = get_results(c_values)
result_one = out[0]
result_two = out[1]
I'm not sure exactly what you're trying to accomplish with the shifted results, but in terms of concurrency, you should check out concurrent.futures to execute parallel tasks.我不确定你想用转移的结果完成什么,但就并发而言,你应该检查concurrent.futures来执行并行任务。 Also, take a look at functools.partial to create a partial object - essentially a function with pre-filled args / kwargs.另外,看看functools.partial来创建一个部分对象 - 本质上是一个带有预填充 args / kwargs 的函数。 Here's an example:下面是一个例子:
import concurrent.futures
from functools import partial
import numpy as np
def map_processes(func, _iterable):
with concurrent.futures.ProcessPoolExecutor() as executor:
result = executor.map(func, _iterable)
return result
def function(x, a, b, c):
return a * x**2 + b * (x + c)
if __name__ == "__main__":
base_func = partial(function, np.linspace(-1, 1, 50), 1, 1)
print(list(map_processes(base_func, np.array([1, 2, 3]))))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.