[英]How to properly create and run concurrent tasks using python's asyncio module?
I am trying to properly understand and implement two concurrently running Task
objects using Python 3's relatively new asyncio
module.我正在尝试使用 Python 3 的相对较新的
asyncio
模块正确理解和实现两个同时运行的Task
对象。
In a nutshell, asyncio seems designed to handle asynchronous processes and concurrent Task
execution over an event loop.简而言之,asyncio 似乎旨在通过事件循环处理异步进程和并发
Task
执行。 It promotes the use of await
(applied in async functions) as a callback-free way to wait for and use a result, without blocking the event loop.它提倡使用
await
(应用于异步函数)作为等待和使用结果的无回调方式,而不会阻塞事件循环。 (Futures and callbacks are still a viable alternative.) (期货和回调仍然是一个可行的选择。)
It also provides the asyncio.Task()
class, a specialized subclass of Future
designed to wrap coroutines.它还提供了
asyncio.Task()
class,这是Future
的专门子类,旨在包装协程。 Preferably invoked by using the asyncio.ensure_future()
method.最好使用
asyncio.ensure_future()
方法调用。 The intended use of asyncio tasks is to allow independently running tasks to run 'concurrently' with other tasks within the same event loop. asyncio 任务的预期用途是允许独立运行的任务与同一事件循环中的其他任务“同时”运行。 My understanding is that
Tasks
are connected to the event loop which then automatically keeps driving the coroutine between await
statements.我的理解是
Tasks
连接到事件循环,然后自动在await
语句之间继续驱动协程。
I like the idea of being able to use concurrent Tasks without needing to use one of the Executor
classes, but I haven't found much elaboration on implementation.我喜欢能够使用并发任务而不需要使用其中一个
Executor
类的想法,但我还没有找到关于实现的详细说明。
This is how I'm currently doing it:这就是我目前的做法:
import asyncio
print('running async test')
async def say_boo():
i = 0
while True:
await asyncio.sleep(0)
print('...boo {0}'.format(i))
i += 1
async def say_baa():
i = 0
while True:
await asyncio.sleep(0)
print('...baa {0}'.format(i))
i += 1
# wrap in Task object
# -> automatically attaches to event loop and executes
boo = asyncio.ensure_future(say_boo())
baa = asyncio.ensure_future(say_baa())
loop = asyncio.get_event_loop()
loop.run_forever()
In the case of trying to concurrently run two looping Tasks, I've noticed that unless the Task has an internal await
expression, it will get stuck in the while
loop, effectively blocking other tasks from running (much like a normal while
loop).在尝试同时运行两个循环任务的情况下,我注意到除非任务具有内部
await
表达式,否则它将卡在while
循环中,从而有效地阻止其他任务运行(很像普通的while
循环)。 However, as soon the Tasks have to (a)wait, they seem to run concurrently without an issue.但是,一旦任务必须(a)等待,它们似乎可以同时运行而没有问题。
Thus, the await
statements seem to provide the event loop with a foothold for switching back and forth between the tasks, giving the effect of concurrency.因此,
await
语句似乎为事件循环提供了一个在任务之间来回切换的立足点,从而产生并发的效果。
Example output with internal await
:带有内部
await
的示例 output :
running async test
...boo 0
...baa 0
...boo 1
...baa 1
...boo 2
...baa 2
Example output without internal await
:示例 output没有内部
await
:
...boo 0
...boo 1
...boo 2
...boo 3
...boo 4
Does this implementation pass for a 'proper' example of concurrent looping Tasks in asyncio
?此实现是否通过了
asyncio
中并发循环任务的“正确”示例?
Is it correct that the only way this works is for a Task
to provide a blocking point ( await
expression) in order for the event loop to juggle multiple tasks?唯一可行的方法是让
Task
提供阻塞点( await
表达式)以便事件循环处理多个任务,这是否正确?
Yes, any coroutine that's running inside your event loop will block other coroutines and tasks from running, unless it是的,任何在您的事件循环中运行的协程都会阻止其他协程和任务运行,除非它
yield from
or await
(if using Python 3.5+).yield from
或await
调用另一个协程(如果使用 Python 3.5+)。 This is because asyncio
is single-threaded;这是因为
asyncio
是单线程的; the only way for the event loop to run is for no other coroutine to be actively executing.事件循环运行的唯一方法是没有其他协程主动执行。 Using
yield from
/ await
suspends the coroutine temporarily, giving the event loop a chance to work.使用
yield from
/ await
暂时挂起协程,让事件循环有机会工作。
Your example code is fine, but in many cases, you probably wouldn't want long-running code that isn't doing asynchronous I/O running inside the event loop to begin with.您的示例代码很好,但在许多情况下,您可能不希望长时间运行的代码不执行在事件循环内运行的异步 I/O。 In those cases, it often makes more sense to use
asyncio.loop.run_in_executor
to run the code in a background thread or process.在这些情况下,使用
asyncio.loop.run_in_executor
在后台线程或进程中运行代码通常更有意义。 ProcessPoolExecutor
would be the better choice if your task is CPU-bound, ThreadPoolExecutor
would be used if you need to do some I/O that isn't asyncio
-friendly.如果您的任务受 CPU 限制,
ProcessPoolExecutor
将是更好的选择,如果您需要执行一些非asyncio
友好的 I/O,则将使用ThreadPoolExecutor
。
Your two loops, for example, are completely CPU-bound and don't share any state, so the best performance would come from using ProcessPoolExecutor
to run each loop in parallel across CPUs:例如,您的两个循环完全受 CPU 限制并且不共享任何状态,因此最好的性能来自使用
ProcessPoolExecutor
跨 CPU 并行运行每个循环:
import asyncio
from concurrent.futures import ProcessPoolExecutor
print('running async test')
def say_boo():
i = 0
while True:
print('...boo {0}'.format(i))
i += 1
def say_baa():
i = 0
while True:
print('...baa {0}'.format(i))
i += 1
if __name__ == "__main__":
executor = ProcessPoolExecutor(2)
loop = asyncio.get_event_loop()
boo = loop.run_in_executor(executor, say_boo)
baa = loop.run_in_executor(executor, say_baa)
loop.run_forever()
You don't necessarily need a yield from x
to give control over to the event loop.您不一定需要
yield from x
的yield from x
来控制事件循环。
In your example, I think the proper way would be to do a yield None
or equivalently a simple yield
, rather than a yield from asyncio.sleep(0.001)
:在您的示例中,我认为正确的方法是执行
yield None
或等效的简单yield
,而不是yield from asyncio.sleep(0.001)
的yield from asyncio.sleep(0.001)
:
import asyncio
@asyncio.coroutine
def say_boo():
i = 0
while True:
yield None
print("...boo {0}".format(i))
i += 1
@asyncio.coroutine
def say_baa():
i = 0
while True:
yield
print("...baa {0}".format(i))
i += 1
boo_task = asyncio.async(say_boo())
baa_task = asyncio.async(say_baa())
loop = asyncio.get_event_loop()
loop.run_forever()
Coroutines are just plain old Python generators.协程只是普通的旧 Python 生成器。 Internally, the
asyncio
event loop keeps a record of these generators and calls gen.send()
on each of them one by one in a never ending loop.在内部,
asyncio
事件循环会保存这些生成器的记录,并在永无止境的循环中对每个生成器调用gen.send()
。 Whenever you yield
, the call to gen.send()
completes and the loop can move on.无论何时
yield
,对gen.send()
的调用gen.send()
完成并且循环可以继续。 (I'm simplifying it; take a look around https://hg.python.org/cpython/file/3.4/Lib/asyncio/tasks.py#l265 for the actual code) (我正在简化它;查看https://hg.python.org/cpython/file/3.4/Lib/asyncio/tasks.py#l265以获得实际代码)
That said, I would still go the run_in_executor
route if you need to do CPU intensive computation without sharing data.也就是说,如果您需要在不共享数据的情况下进行 CPU 密集型计算,我仍然会选择
run_in_executor
路线。
The functions asyncio.ensure_future
and asyncio.get_event_loop
are deprecated in Python 3.10.在 Python 3.10 中不推荐使用
asyncio.ensure_future
和asyncio.get_event_loop
函数。
You can run the two coroutines say_boo
and say_baa
concurrently through asyncio.create_task
:您可以通过
asyncio.create_task
同时运行两个协程say_boo
和say_baa
:
async def main():
boo = asyncio.create_task(say_boo())
baa = asyncio.create_task(say_baa())
await boo
await baa
asyncio.run(main())
You can also use asyncio.gather
你也可以使用
asyncio.gather
async def main():
await asyncio.gather(say_boo(), say_baa())
asyncio.run(main())
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.