[英]Passing argument via executor.map
How can I send another argument via executor.map function?如何通过 executor.map function 发送另一个参数? See an example of code:
查看代码示例:
from concurrent.futures import ProcessPoolExecutor
from dask.dataframe import read_csv
def apply(row_of_small_df):
# Here there is no access to big_df
return
def main():
small_df = read_csv('...')
big_df = read_csv('...')
with ProcessPoolExecutor() as executor:
results = executor.map(apply, small_df.iterrows())
for result in results:
pass
if __name__ == '__main__':
main()
Another alternative is using functools.partial
:另一种选择是使用
functools.partial
:
Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords.返回一个新的部分 object,它在调用时的行为类似于使用位置 arguments 参数和关键字 arguments 关键字调用的 func。 If more arguments are supplied to the call, they are appended to args.
如果向调用提供了更多 arguments,它们将附加到 args。 If additional keyword arguments are supplied, they extend and override keywords.
如果提供了额外的关键字 arguments,它们会扩展和覆盖关键字。
from functools import partial
def apply(big_df, row_of_small_df):
# requires big_df to be passed in
return
def main():
small_df = read_csv('...')
big_df = read_csv('...')
apply_with_big_df = partial(apply, big_df)
with ProcessPoolExecutor() as executor:
results = executor.map(apply_with_big_df, small_df.iterrows())
for result in results:
pass
use a lambda:使用 lambda:
#...
def apply(big_df, row_of_small_df):
pass
#...
results = executor.map(lambda row_of_small: apply(big_df, row_of_small), small_df.iterrows())
#...
Use the zip function like so像这样使用 zip function
results = executor.map(apply, zip(big_df.iterrows(), small_df.iterrows()))
The function should now be function 现在应该是
def apply(params):
big, small = params
# your code
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.