使用字典调用python多处理映射函数

Question

我可以使用imap执行以下操作：

import multiprocessing

def function(a):
    return a**2

b = [1,2,3,4,5,6,7,8,9,10]

pool = multiprocessing.Pool(processes=4)
vals = pool.imap(function, b)

for i in vals:
    print(i)

但是，如果我想传入一个字典，可以这样做：

import multiprocessing

def function(name, a):
    print("{}'s number squared:".format(name))
    return a**2

b = dict(bob=1, dave=2, stan=3)

pool = multiprocessing.Pool(processes=4)
vals = pool.imap(function, b)

for i in vals:
    print(i)

Answer 1

您可以传递字典，但对字典的map操作仅提供字典键。 如果您想要字典中的项目，请使用dict.items ，例如：

import multiprocessing
def function(item):
    name, a = item
    print("{}'s number squared:".format(name))
    return a**2

if __name__ == '__main__': # you must guard your multiprocessing calls!

    b = dict(bob=1, dave=2, stan=3)

    with multiprocessing.Pool(processes=4) as pool:
        vals = pool.imap(function, list(b.items()))

        for i in vals:
            print(i)

或者，如果您希望相同的函数签名起作用，则可以使用imap ：

import multiprocessing
def function(name, a):
    print("{}'s number squared:".format(name))
    return a**2

if __name__ == '__main__': # you must guard your multiprocessing calls!

    b = dict(bob=1, dave=2, stan=3)

    with multiprocessing.Pool(processes=4) as pool:
        vals = pool.starmap(function, list(b.items()))

    for i in vals:
        print(i)

另一种方法是使用全局变量，将 dict 直接传递给pool.map ，虽然它可能有点难看：

import multiprocessing
def function(name):
    a = b[name]
    print("{}'s number squared:".format(name))
    return a**2

b = dict(bob=1, dave=2, stan=3)

if __name__ == '__main__': # you must guard your multiprocessing calls!


    with multiprocessing.Pool(processes=4) as pool:
        vals = pool.imap(function, b)

        for i in vals:
            print(i)

注意，我使用了muliprocessing.Pool作为上下文管理器，所以当你离开with块时，无论发生什么，池都将被保证关闭。 这是最佳做法。

Answer 2

您不能传递字典，但可以传递字典列表，如下所示：

import multiprocessing

def function(input_dictionary):
    print("{}'s number squared:".format(input_dictionary['name']))
    return input_dictionary['number']**2

b = [ {'name' : 'bob', 'number' : 1},
      {'name' : 'dave', 'number' : 2},
      {'name' : 'stan', 'number' : 3}, ]


pool = multiprocessing.Pool(processes=4)
vals = pool.imap(function, b)

for i in vals:
    print(i)