[英]asyncio.Queue producer-consumer flow cannot handle exception when consumers contain in a named list
Working on a producer-consumer flow based on the asyncio.Queue
.处理基于
asyncio.Queue
的生产者-消费者流。
Codes below take reference from this answer and this blog .下面的代码参考了这个答案和这个博客。
import asyncio
async def produce(q: asyncio.Queue, t):
asyncio.create_task(q.put(t))
print(f'Produced {t}')
async def consume(q: asyncio.Queue):
while True:
res = await q.get()
if res > 2:
print(f'Cannot consume {res}')
raise ValueError(f'{res} too big')
print(f'Consumed {res}')
q.task_done()
async def shutdown(loop, signal=None):
tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
print(f"Cancelling {len(tasks)} outstanding tasks")
[task.cancel() for task in tasks]
def handle_exception(loop, context):
msg = context.get("exception", context["message"])
print(f"Caught exception: {msg}")
asyncio.create_task(shutdown(loop))
async def main():
queue = asyncio.Queue()
loop = asyncio.get_event_loop()
loop.set_exception_handler(handle_exception)
[asyncio.create_task(consume(queue)) for _ in range(1)]
# consumers = [asyncio.create_task(consume(queue)) for _ in range(1)]
try:
for i in range(6):
await asyncio.create_task(produce(queue, i))
await queue.join()
except asyncio.exceptions.CancelledError:
print('Cancelled')
asyncio.run(main())
When wrapping the consumers like above (without a naming list), the output is as expected:当像上面那样包装消费者时(没有命名列表),output 符合预期:
Produced 0
Consumed 0
Produced 1
Consumed 1
Produced 2
Consumed 2
Produced 3
Cannot consume 3
Caught exception: 3 too big
Produced 4
Cancelling 2 outstanding tasks
Cancelled
But when giving the consumer list a name, which means change the code inside main()
like this:但是当给消费者列表一个名字时,这意味着改变
main()
中的代码,如下所示:
async def main():
# <-- snip -->
# [asyncio.create_task(consume(queue)) for _ in range(1)]
consumers = [asyncio.create_task(consume(queue)) for _ in range(1)]
# <-- snip -->
The program gets stuck like this:程序像这样卡住:
Produced 0
Consumed 0
Produced 1
Consumed 1
Produced 2
Consumed 2
Produced 3
Cannot consume 3
Produced 4
Produced 5 # <- stuck here, have to manually stop by ^C
It seems like the producer
still keeps producing so that the items in the queue
keeps growing after the ValueError
raised.似乎
producer
仍在继续生产,以便在引发ValueError
后queue
中的项目不断增长。 The handle_exception
never get called. handle_exception
永远不会被调用。 And the program gets stuck at the await queue.join()
.并且程序卡在
await queue.join()
。
But why giving a name to the consumers list would change the behavior of the code?但是为什么给消费者列表命名会改变代码的行为呢? Why the
handle_exception
never get called after consumers list being named?为什么在消费者列表被命名后,
handle_exception
永远不会被调用?
TL;DR Don't use set_exception_handler
to handle exception in tasks. TL;DR 不要使用
set_exception_handler
来处理任务中的异常。 Instead, add the requisite try: ... except: ...
in the coroutine itself.相反,在协程本身中添加必要的
try: ... except: ...
The problem is in the attempt to use set_exception_handler
to handle exceptions.问题在于尝试使用
set_exception_handler
来处理异常。 That function is a last-ditch attempt to detect an exception that has passed through all the way to the event loop, most likely as the result of a bug in the program. function 是检测一直到事件循环的异常的最后尝试,很可能是程序中的错误的结果。 If a callback added by
loop.call_soon
or loop.call_at
etc. raises an exception (and doesn't catch it), the handler installed by set_exception_handler
will be consistently invoked.如果由
loop.call_soon
或loop.call_at
等添加的回调引发异常(并且没有捕获它),则set_exception_handler
安装的处理程序将被一致地调用。
With a task things are more nuanced: a task drives a coroutine to completion and, once done, stores its result , making it available to anyone who awaits the task, to callbacks installed by add_done_callback
, but also to any call that invokes result()
on the task.有了任务,事情就更微妙了:任务驱动协程完成,一旦完成,就会存储它的结果,让等待任务的任何人都可以使用它,可以使用
add_done_callback
安装的回调,也可以用于任何调用result()
的调用在任务上。 (All this is mandated by the contract of Future
, which Task
is a subclass of.) When the coroutine raises an unhandled exception, this exception is just another result: when someone awaits the task or invokes result()
, the exception will be (re-)raised then and there. (所有这一切都由
Future
的合约规定, Task
是它的子类。)当协程引发未处理的异常时,此异常只是另一个结果:当有人等待任务或调用result()
时,异常将是 (重新)当时和那里加注。
This leads to the difference between naming and not naming the task objects.这导致命名和不命名任务对象之间的差异。 If you don't name them, they will be destroyed as soon as the event loop is done executing them.
如果你不命名它们,一旦事件循环完成执行它们,它们就会被销毁。 At the point of their destruction, Python will notice that no one has ever accessed their result and will pass it to the exception handler.
在它们被销毁的时候,Python 会注意到没有人访问过它们的结果并将其传递给异常处理程序。 On the other hand, if you store them in a variable, they won't be destroyed as long as they're referenced by the variable and there will be no reason to call the event loop handler: as far as Python is concerned, you might decide to call
.result()
on the objects at any point, access the exception and handle it as appropriate for your program.另一方面,如果将它们存储在变量中,只要它们被变量引用,它们就不会被销毁,并且没有理由调用事件循环处理程序:就 Python 而言,您可能决定在任何时候对对象调用
.result()
,访问异常并根据您的程序对其进行处理。
To fix the issue, just handle the exception yourself by adding a try: ... except: ...
block around the body of the coroutine.要解决此问题,只需通过在协程主体周围添加
try: ... except: ...
块来自己处理异常。 If you don't control the coroutine, you can use add_done_callback()
to detect the exception instead.如果你不控制协程,你可以使用
add_done_callback()
来检测异常。
It's not about the named list.这与命名列表无关。 Your example can be simplified to:
您的示例可以简化为:
asyncio.create_task(consume(queue))
# consumer = asyncio.create_task(consume(queue))
The point here is in the Task
object that the function create_task
returns.这里的重点是 function
create_task
返回的Task
object。 In one case, it is destroyed, but in the other not.在一种情况下,它被破坏了,但在另一种情况下没有。 Good answers have been given here and here
在这里和这里都给出了很好的答案
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.