[英]Set max concurrency with asyncio.create_subprocess_exec
I need to run a program about 500 times with different inputs. 我需要使用不同的输入运行一个程序大约500次。 I'd like to use
asyncio.create_subprocess_exec
and want to limit the number of processes running at the same time so as not to clog up the machine. 我想使用
asyncio.create_subprocess_exec
并希望限制同时运行的进程数,以免阻塞计算机。 Is there a way to set the concurrency level? 有没有办法设置并发级别? For example, I'd expect something like
AbstractEventLoop.set_max_tasks
. 例如,我期望像
AbstractEventLoop.set_max_tasks
这样的东西。
As suggested by @AndrewSvetlov, you can use an asyncio.Semaphore
to enforce the limit: 如@AndrewSvetlov所建议 ,您可以使用
asyncio.Semaphore
来强制执行限制:
async def run_program(input):
p = await asyncio.create_subprocess_exec(...)
# ... communicate with the process ...
p.terminate()
return something_useful
async def run_throttled(input, sem):
async with sem:
result = await run_program(input)
return result
LIMIT = 10
async def many_programs(inputs):
sem = asyncio.Semaphore(LIMIT)
results = await asyncio.gather(
*[run_throttled(input, sem) for input in inputs])
# ...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.