[英]Running different Python functions in separate CPUs
Using multiprocessing.pool
I can split an input list for a single function to be processed in parallel along multiple CPUs. 使用multiprocessing.pool
我可以为一个要分割的功能拆分一个输入列表,以便在多个CPU上并行处理。 Like this: 像这样:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
pool = Pool(processes=4)
results = pool.map(f, range(100))
pool.close()
pool.join()
However, this does not allow to run different functions on different processors. 但是,这不允许在不同的处理器上运行不同的功能。 If I want to do something like this, in parallel / simultaneously: 如果我想并行/同时执行以下操作:
foo1(args1) --> Processor1
foo2(args2) --> Processor2
How can this be done? 如何才能做到这一点?
Edit: After Darkonaut remarks, I do not care about specifically assigning foo1 to Processor number 1. It can be any processor as chosen by the OS. 编辑:在Darkonaut发言之后,我不关心将foo1专门分配给处理器编号1。它可以是操作系统选择的任何处理器。 I am just interested in running independent functions in different/ parallel Processes. 我只是对在不同/并行进程中运行独立功能感兴趣。 So rather: 所以:
foo1(args1) --> process1
foo2(args2) --> process2
I usually find it easiest to use the concurrent.futures module for concurrency. 我通常发现最简单的方法是使用并发模块 。 You can achieve the same with multiprocessing
, but concurrent.futures
has (IMO) a much nicer interface. 您可以通过multiprocessing
实现相同的功能,但是concurrent.futures
具有(IMO)更好的接口。
Your example would then be: 您的示例将是:
from concurrent.futures import ProcessPoolExecutor
def foo1(x):
return x * x
def foo2(x):
return x * x * x
if __name__ == '__main__':
with ProcessPoolExecutor(2) as executor:
# these return immediately and are executed in parallel, on separate processes
future_1 = executor.submit(foo1, 1)
future_2 = executor.submit(foo2, 2)
# get results / re-raise exceptions that were thrown in workers
result_1 = future_1.result() # contains foo1(1)
result_2 = future_2.result() # contains foo2(2)
If you have many inputs, it is better to use executor.map
with the chunksize
argument instead: 如果输入很多,最好将executor.map
与chunksize
参数一起使用:
from concurrent.futures import ProcessPoolExecutor
def foo1(x):
return x * x
def foo2(x):
return x * x * x
if __name__ == '__main__':
with ProcessPoolExecutor(4) as executor:
# these return immediately and are executed in parallel, on separate processes
future_1 = executor.map(foo1, range(10000), chunksize=100)
future_2 = executor.map(foo2, range(10000), chunksize=100)
# executor.map returns an iterator which we have to consume to get the results
result_1 = list(future_1) # contains [foo1(x) for x in range(10000)]
result_2 = list(future_2) # contains [foo2(x) for x in range(10000)]
Note that the optimal values for chunksize
, the number of processes, and whether process-based concurrency actually leads to increased performance depends on many factors: 请注意, chunksize
的最佳值,进程数以及基于进程的并发是否真正导致性能提高取决于许多因素:
The runtime of foo1
/ foo2
. foo1
/ foo2
的运行时。 If they are extremely cheap (as in this example), the communication overhead between processes might dominate the total runtime. 如果它们非常便宜(如本例所示),则进程之间的通信开销可能会占总运行时间的主导地位。
Spawning a process takes time, so the code inside with ProcessPoolExecutor
needs to run long enough for this to amortize. 生成一个进程需要花费时间,因此with ProcessPoolExecutor
内部的代码需要运行足够长的时间才能摊销。
The actual number of physical processors in the machine you are running on. 您正在运行的计算机上的物理处理器的实际数量。
Whether your application is IO bound or compute bound. 您的应用程序是IO绑定还是计算绑定。
Whether the functions you use in foo
are already parallelized (such as some np.linalg
solvers, or scikit-learn
estimators). 您在foo
中使用的函数是否已经并行化(例如某些np.linalg
求解器或scikit-learn
估计器)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.