通过 executor.map 传递参数

Question

如何通过 executor.map function 发送另一个参数？ 查看代码示例：

from concurrent.futures import ProcessPoolExecutor
from dask.dataframe import read_csv


def apply(row_of_small_df):
    # Here there is no access to big_df
    return


def main():
    small_df = read_csv('...')
    big_df = read_csv('...')
    with ProcessPoolExecutor() as executor:
        results = executor.map(apply, small_df.iterrows())
        for result in results:
            pass


if __name__ == '__main__':
    main()

Answer 1

另一种选择是使用functools.partial ：

返回一个新的部分 object，它在调用时的行为类似于使用位置 arguments 参数和关键字 arguments 关键字调用的 func。 如果向调用提供了更多 arguments，它们将附加到 args。 如果提供了额外的关键字 arguments，它们会扩展和覆盖关键字。

from functools import partial

def apply(big_df, row_of_small_df):
    # requires big_df to be passed in
    return

def main():
    small_df = read_csv('...')
    big_df = read_csv('...')

    apply_with_big_df = partial(apply, big_df)

    with ProcessPoolExecutor() as executor:
        results = executor.map(apply_with_big_df, small_df.iterrows())
        for result in results:
            pass

Answer 2

使用 lambda：

#...
def apply(big_df, row_of_small_df):
    pass
#...
results = executor.map(lambda row_of_small: apply(big_df, row_of_small), small_df.iterrows())
#...

Answer 3

像这样使用 zip function

results = executor.map(apply, zip(big_df.iterrows(), small_df.iterrows()))

function 现在应该是

def apply(params):
    big, small = params
    # your code