[英]How to use run_in_executor right?
I try to use run_in_executor
and have some questions. 我尝试使用
run_in_executor
并提出一些问题。 Here is code (basically copypast from docs) 这是代码(基本上来自docs的copypast)
import asyncio
import concurrent.futures
def cpu_bound(val):
# CPU-bound operations will block the event loop:
# in general it is preferable to run them in a
# process pool.
print(f'Start task: {val}')
sum(i * i for i in range(10 ** 7))
print(f'End task: {val}')
async def async_task(val):
print(f'Start async task: {val}')
while True:
print(f'Tick: {val}')
await asyncio.sleep(1)
async def main():
loop = asyncio.get_running_loop()
## Options:
for i in range(5):
loop.create_task(async_task(i))
# 1. Run in the default loop's executor:
# for i in range(10):
# loop.run_in_executor(
# None, cpu_bound, i)
# print('default thread pool')
# 2. Run in a custom thread pool:
# with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
# for i in range(10):
# loop.run_in_executor(
# pool, cpu_bound, i)
# print('custom thread pool')
# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor(max_workers = 10) as pool:
for i in range(10):
loop.run_in_executor(
pool, cpu_bound, i)
print('custom process pool')
while True:
await asyncio.sleep(1)
asyncio.run(main())
Case 1: run_in_executor
where executor
is None
: async_task
's execute in the same time as cpu_bound
's execute. 情况1:
run_in_executor
其中executor
是None
: async_task
的执行与cpu_bound
的执行同时执行。
In other cases async_task
's will execute after cpu_bound
's are done. 在其他情况下,
async_task
将在cpu_bound
完成后执行。 I thought when we use ProcessPoolExecutor
tasks shouldn't block loop. 我
ProcessPoolExecutor
我们使用ProcessPoolExecutor
任务不应该阻塞循环。 Where am I wrong? 我哪里错了?
In other cases
async_task
's will execute aftercpu_bound
's are done.在其他情况下,
async_task
将在cpu_bound
完成后执行。 I thought when we useProcessPoolExecutor
tasks shouldn't block loop.我
ProcessPoolExecutor
我们使用ProcessPoolExecutor
任务不应该阻塞循环。 Where am I wrong?我哪里错了?
The problem is that with XXXPoolExecutor()
shuts down the pool at the end of the with
block. 问题是
with XXXPoolExecutor()
在with
块的末尾关闭池。 Pool shutdown waits for the pending tasks to finish, which blocks the event loop and is incompatible with asyncio. 池关闭等待挂起的任务完成,这会阻止事件循环并与asyncio不兼容。 Since your first variant doesn't involve a
with
statement, it doesn't have this issue. 由于您的第一个变体不涉及
with
语句,因此它没有此问题。
The solution is simply to remove the with
statement and create the pool once (for example at top-level or in main()
), and just use it in the function. 解决方案只是删除
with
语句并创建一次池(例如在顶层或在main()
),并在函数中使用它。 If you want to, you can explicitly shut down the pool by calling pool.shutdown()
after asyncio.run()
has completed. 如果您愿意,可以在
asyncio.run()
完成后通过调用pool.shutdown()
显式关闭池。
Also note that you are never awaiting the futures returned by loop.run_in_executor
. 另请注意,您永远不会等待
loop.run_in_executor
返回的期货。 This is an error and asyncio will probably warn you of it; 这是一个错误,asyncio可能会警告你; you should probably collect the returned values in a list and await them with something like
results = await asyncio.gather(*tasks)
. 你可能应该在列表中收集返回的值,并等待
results = await asyncio.gather(*tasks)
。 This will not only collect the results, but also make sure that the exceptions that occur in the off-thread functions get correctly propagated to your code rather than dropped. 这不仅会收集结果,还会确保在线程外函数中发生的异常正确传播到您的代码而不是丢弃。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.