如何在 python 中多处理 function

Question

我的代码：

for batchList in all_List:
    result = getBatch_filter(batchList, entTotal, tripleList, neg_rate)

def getBatch_filter(batchList, entTotal, tripleList, neg_rate):
    result = []
    for item in batchList:
        a = corrupt_a(item, entTotal, tripleList, neg_rate)
        b = corrupt_b(item, entTotal, tripleList, neg_rate)
        result += a
        result += b
    return result

其中 batchList, a, b 是嵌套列表，例如： [[1,2,3],[3,4,1,4,6],[6,9,10,11]...]我想多进程function getBatch_filter中的for循环。 我试过 Pool，但它运行整个文件而不是 function。 任何帮助，将不胜感激。

Answer 1

看这个例子：

from multiprocessing import Pool
def heavy_func(key):
    #do some heavy computation on each key 
    output = key**2
    return key, output 

output_data ={}     #<--this dict will store the results
keys = [1,5,7,8,10] #<--compute heavy_func over all the values of keys
with Pool(processes=40) as pool:
    for i in pool.imap_unordered(heavy_func, keys):
        output_data[i[0]] = i[1]

因此，在您的情况下，您将执行以下操作：

from multiprocessing import Pool
# assuming  entTotal, tripleList, neg_rate are globals and defined here
def getBatch_filter(batchList):
    result = []
    for item in batchList:
        a = corrupt_a(item, entTotal, tripleList, neg_rate)
        b = corrupt_b(item, entTotal, tripleList, neg_rate)
        result += a
        result += b
    return batchList, result


output_data ={}     #<--this dict will store the results
keys = all_List #<--compute heavy_func over all the values of keys
with Pool(processes=40) as pool:
    for i in pool.imap_unordered(getBatch_filter, keys):
        output_data[i[0]] = i[1]