![](/img/trans.png)
[英]asyncio.Queue producer-consumer flow cannot handle exception when consumers contain in a named list
[英]Notify producer to stop producing when consumer raise exception in the asyncio.Queue produce-consume flow
基于asyncio.Queue
的模拟生产-消费流程:
import asyncio
async def produce(q: asyncio.Queue, task):
asyncio.create_task(q.put(task))
print(f'Produced {task}')
async def consume(q: asyncio.Queue):
while True:
task = await q.get()
if task > 2:
print(f'Cannot consume {task}')
raise ValueError(f'{task} too big')
print(f'Consumed {task}')
q.task_done()
async def main():
queue = asyncio.Queue()
consumers = [asyncio.create_task(consume(queue)) for _ in range(2)]
for i in range(10):
await asyncio.create_task(produce(queue, i))
await asyncio.wait([queue.join(), *consumers],
return_when=asyncio.FIRST_COMPLETED)
asyncio.run(main())
output 是:
Produced 0
Consumed 0
Produced 1
Consumed 1
Produced 2
Consumed 2
Produced 3
Cannot consume 3
Produced 4
Cannot consume 4
Produced 5
Produced 6
Produced 7
Produced 8
Produced 9
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<consume() done, defined at test.py:9> exception=ValueError('3 too big')>
Traceback (most recent call last):
File "test.py", line 14, in consume
raise ValueError(f'{task} too big')
ValueError: 3 too big
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<consume() done, defined at test.py:9> exception=ValueError('4 too big')>
Traceback (most recent call last):
File "test.py", line 14, in consume
raise ValueError(f'{task} too big')
ValueError: 4 too big
有没有办法在消费者提出异常后通知生产者停止生产?
上面的代码使用了多个生产者。 如果“通知”机制只能在单生产者模式下工作,也是可以接受的。
灵感来自user4815162342的建议
设置全局 Boolean 变量以防消费者异常,并在生产者中检查
import asyncio
stop = False
async def single_produce(q: asyncio.Queue):
global stop
for task in range(10):
await asyncio.sleep(0.001)
if stop:
break
await q.put(task)
print(f'Produced {task}')
async def multi_produce(q: asyncio.Queue, task):
await asyncio.sleep(0.001)
await q.put(task)
print(f'Produced {task}')
async def consume(q: asyncio.Queue):
global stop
while True:
task = await q.get()
if task > 2:
stop = True
print(f'Cannot consume {task}')
raise ValueError(f'{task} too big')
print(f'Consumed {task}')
q.task_done()
async def main(mode):
global stop
queue = asyncio.Queue(1)
consumers = [asyncio.create_task(consume(queue)) for _ in range(2)]
if mode == 'single':
print('single producer')
await asyncio.create_task(single_produce(queue))
elif mode == 'multiple':
print('multiple producers')
for i in range(10):
if stop:
break
await asyncio.create_task(multi_produce(queue, i))
await asyncio.wait([queue.join(), *consumers],
return_when=asyncio.FIRST_COMPLETED)
asyncio.run(main('single'))
# asyncio.run(main('multiple'))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.