[英]Multiprocessing: How to use pool.map on a list and function with arguments?
I have some misunderstandings with multiprocessing and map function. 我对多处理和地图功能有一些误解。
I'll try to describe briefly: 我将简要介绍一下:
Firstly, I have an list, for instance: 首先,我有一个列表,例如:
INPUT_MAGIC_DATA_STRUCTURE = [
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
['https://github.com', 'Owner', 'Repo', '', '', '0', '0'],
]
Also I have method, which currently parsing this list using specific internal logic: 我也有方法,目前使用特定的内部逻辑解析此列表:
def parse(api_client1, api_client2):
for row in INPUT_MAGIC_DATA_STRUCTURE:
parsed_repo_row = ... (some logic with row)
OUTPUT_MAGIC_DATA_STRUCTURE.append(parsed_repo_row)
Finally, I've red that there is some variants to make it async instead of for
. 最后,我发现有一些变种可以使它成为异步而不是
for
。
from multiprocessing import Pool
pool = Pool(10)
pool.map(<???>, INPUT_MAGIC_STRUCTURE)
???
– I cannot understand how to transfer my parse()
from for row in INPUT_MAGIC_DATA_STRUCTURE
as a first argument to pool.map()
and transfer all its arguments — api_client1, api_client2. - 我无法理解如何将我的
parse()
从for row in INPUT_MAGIC_DATA_STRUCTURE
中的for row in INPUT_MAGIC_DATA_STRUCTURE
作为第一个参数传递给pool.map()
并传递其所有参数 - api_client1,api_client2。
Could you help me? 你可以帮帮我吗?
Thanks in advance. 提前致谢。
UPD: UPD:
I've already made: 我已经做过:
pool = Pool(10)
pool.map(parse(magic_parser, magic_staff), INPUT_MAGIC_DATA_STRUCTURE)
Anyway, when interpreter comes to the second line it stops and makes only one instance of parse() method (I see the logging output of parsed rows: 1 , 2 , 3 , 4 , 5 – one by one). 无论如何,当解释器到达第二行时它停止并只生成一个parse()方法的实例(我看到解析行的日志输出:1,2,3,4,5 - 逐个)。
Put (some logic with row)
in a function: 在函数中放入
(some logic with row)
:
def row_logic(row):
return result
Pass the function to Pool.map
: 将函数传递给
Pool.map
:
pool = Pool(10)
pool.map(row_logic, INPUT_MAGIC_DATA_STRUCTURE)
We'll, in python it's not that easy. 我们在python中并不那么容易。 You need to map your rows to your parse per row function.
您需要将行映射到每行解析函数。 Look at this link: https://gist.github.com/baojie/6047780
请看这个链接: https : //gist.github.com/baojie/6047780
from multiprocessing import Process
def parse_row(row):
(some logic with row)
def dispatch_job(rows, parse_row):
for row in rows:
p = Process(target=parse_row, args=(row,))
p.start()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.