通過 executor.map 傳遞參數

Question

如何通過 executor.map function 發送另一個參數？ 查看代碼示例：

from concurrent.futures import ProcessPoolExecutor
from dask.dataframe import read_csv


def apply(row_of_small_df):
    # Here there is no access to big_df
    return


def main():
    small_df = read_csv('...')
    big_df = read_csv('...')
    with ProcessPoolExecutor() as executor:
        results = executor.map(apply, small_df.iterrows())
        for result in results:
            pass


if __name__ == '__main__':
    main()

Answer 1

另一種選擇是使用functools.partial ：

返回一個新的部分 object，它在調用時的行為類似於使用位置 arguments 參數和關鍵字 arguments 關鍵字調用的 func。 如果向調用提供了更多 arguments，它們將附加到 args。 如果提供了額外的關鍵字 arguments，它們會擴展和覆蓋關鍵字。

from functools import partial

def apply(big_df, row_of_small_df):
    # requires big_df to be passed in
    return

def main():
    small_df = read_csv('...')
    big_df = read_csv('...')

    apply_with_big_df = partial(apply, big_df)

    with ProcessPoolExecutor() as executor:
        results = executor.map(apply_with_big_df, small_df.iterrows())
        for result in results:
            pass

Answer 2

使用 lambda：

#...
def apply(big_df, row_of_small_df):
    pass
#...
results = executor.map(lambda row_of_small: apply(big_df, row_of_small), small_df.iterrows())
#...

Answer 3

像這樣使用 zip function

results = executor.map(apply, zip(big_df.iterrows(), small_df.iterrows()))

function 現在應該是

def apply(params):
    big, small = params
    # your code