[英]Best use of multiprocessing Pool for nuanced loop?
I am trying to figure out the best way to make use of python's multiprocessing Pool
. 我试图找出利用python的multiprocessing
Pool
的最佳方法。
I have an n^2 nested for loop that makes a comparison for each bucket combination from a list of buckets. 我有一个n ^ 2嵌套的for循环,用于比较存储桶列表中的每个存储桶组合。
The work that I would like to parallelize using the multiprocessing Pool
is the compare()
function call. 我想使用多处理
Pool
进行并行化的工作是compare()
函数调用。
There are absolutely no shared resources in the compare
function. compare
功能中绝对没有共享资源。 If buckets A and B were comparing while buckets A and C were through another process, it would not matter. 如果存储桶A和B正在比较,而存储桶A和C正在通过另一个过程,则没关系。
I am new to parallel processing but I do understand the basic nature of the multiprocessing Pool
. 我是并行处理的新手,但我确实了解多处理
Pool
的基本性质。 However, I am finding it difficult to implement anything that does what I would like it to do. 但是,我发现很难实现我想做的任何事情。
The reliance of the specific bucket pairs to pass to the function, along with their associated reader_list
seems to be blocking me when I look at any examples of the Pool
. 当我查看
Pool
任何示例时,对特定存储桶对传递给函数的依赖以及与它们相关联的reader_list
似乎使我reader_list
。 I am not necessarily relying on a list that the function gets executed on each index of. 我不一定要依赖该函数在每个索引上执行的列表。
for i in range(0, len(bucket_names) - 1):
bucket1 = bucket_names[i]
for k in range(i+1, len(bucket_names)):
bucket2 = bucket_names[k]
reader_list1 = get_reader_list(bucket1)
reader_list2 = get_reader_list(bucket2)
compare(bucket1, bucket2, reader_list1, reader_list2)
Do you mean you need an example of how to use Pool to parallelize your function? 您的意思是您需要一个如何使用Pool并行化功能的示例吗? Here's an example.
这是一个例子。
import multiprocessing as mp
# Generate your arguments as a list of tuples, using some method that fits your requirements.
# Here is a hard-coded example
arguments = [(bucket1, bucket2), (bucket2, bucket3), (bucket1, bucket3)]
# Create pool given number of logical CPUs you have
pool = mp.Pool(mp.cpu_count())
# Assign work to pool (provide function and list of arguments)
# Results will be list of results
results = pool.starmap(compare, arguments)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.