[英]Is there a way to call an `async` python method from C++?
We have a codebase in python which uses asyncio, and co-routines ( async
methods and await
s), what I'd like to do is to call one of these method from a C++ class which has been pulled into python (using pybind11) 我们在python中有一个代码库,它使用asyncio和协同例程(
async
方法和await
s),我想做的是从C ++类中调用这些方法之一,这个类被拉入python(使用pybind11)
Let's say there is this code: 假设有这样的代码:
class Foo:
async def bar(a, b, c):
# some stuff
return c * a
Assuming that the code is being invoked from python and there is an io loop handling this, at some point, the code drops into C++ land where this bar
method needs to be invoked - how does one await
the result of this in C++? 假设代码是从python中调用的,并且有一个io循环处理它,在某些时候,代码会进入C ++领域,需要调用这个
bar
方法 - 如何在C ++中await
结果呢?
This isn't pybind11, but you can call an async function directly from C. You simply add a callback to the future using add_done_callback. 这不是pybind11,但您可以直接从C调用异步函数。您只需使用add_done_callback向未来添加回调。 I assume pybind11 allows you to call python functions so the steps would be the same:
我假设pybind11允许你调用python函数,所以步骤将是相同的:
https://github.com/MarkReedZ/mrhttp/blob/master/src/mrhttp/internals/protocol.c https://github.com/MarkReedZ/mrhttp/blob/master/src/mrhttp/internals/protocol.c
result = protocol_callPageHandler(self, r->func, request))
Now the result of an async function is a future. 现在,异步功能的结果是未来。 Just like in python you need to call create_task using the resulting future:
就像在python中一样,你需要使用生成的未来调用create_task:
PyObject *task;
if(!(task = PyObject_CallFunctionObjArgs(self->create_task, result, NULL))) return NULL;
And then you need to add a callback using add_done_callback: 然后你需要使用add_done_callback添加回调:
add_done_callback = PyObject_GetAttrString(task, "add_done_callback")
PyObject_CallFunctionObjArgs(add_done_callback, self->task_done, NULL)
self->task_done is a C function registered in python which will be called when the task is done. self-> task_done是在python中注册的C函数,将在任务完成时调用。
It is possible to implement a Python coroutine in C++, but takes some work. 可以在C ++中实现Python协程,但需要一些工作。 You need to do what the interpreter (in static languages the compiler) normally does for you and transform your async function into a state machine.
您需要执行解释器(在编译器的静态语言中)通常为您执行的操作,并将异步函数转换为状态机。 Consider a very simple coroutine:
考虑一个非常简单的协程:
async def coro():
x = foo()
y = await bar()
baz(x, y)
return 42
Invoking coro()
doesn't run any of its code, but it produces an awaitable object which can be started and then resumed multiple times. 调用
coro()
不会运行任何代码,但会产生一个等待的对象,可以启动然后多次恢复。 (But you don't normally see these operations because they are transparently performed by the event loop.) The awaitable can respond in two different ways: by 1) suspending, or by 2) indicating that it is done. (但是你通常不会看到这些操作,因为它们是由事件循环透明地执行的。)等待可以通过两种不同的方式响应:1)暂停,或2)指示它已完成。
Inside a coroutine await
implements suspension. 在协程内
await
实现暂停。 If a coroutine were implemented with a generator, y = await bar()
would desugar to: 如果使用生成器实现协程,则
y = await bar()
将desugar:
# pseudo-code for y = await bar()
_bar_iter = bar().__await__()
while True:
try:
_suspend_val = next(_bar_iter)
except StopIteration as _stop:
y = _stop.value
break
yield _suspend_val
In other words, await
suspends (yields) as long as the awaited object does. 换句话说,只要等待的对象,就
await
暂停(yield)。 The awaited object signals that it's done by raising StopIteration
, and by smuggling the return value inside its value
attribute. 等待的对象表示它是通过提高
StopIteration
并通过走私其value
属性中的返回值来完成的。 If yield-in-a-loop sounds like yield from
, you're exactly right, and that is why await
is often described in terms of yield from
. 如果产量在-一环听起来像是
yield from
,你是完全正确的,这就是为什么await
往往是在来描述yield from
。 However, in C++ we don't have yield
( yet ), so we have to integrate the above into the state machine. 然而,在C ++中,我们没有
yield
( 还 ),所以我们要在上面集成到状态机。
To implement async def
from scratch, we need to have a type that satisfies the following constraints: 要从头开始实现
async def
,我们需要一个满足以下约束的类型:
__await__
method that returns an iterable, which can just be self
; __await__
方法返回一个iterable,它可以只是self
; __iter__
which returns an iterator, which can again be self
; __iter__
返回一个迭代器,它可以再次成为self
; __next__
method whose invocation implements one step of the state machine, with return meaning suspension and raising StopIteration
meaning finishing. __next__
方法,其调用实现状态机的一个步骤,返回意味着暂停并提高StopIteration
意味着完成。 The above coroutine's state machine in __next__
will consist of three states: 在上述协程的状态机
__next__
将包括三种状态:
foo()
sync function foo()
同步函数时 bar()
coroutine for as long as it suspends (propagating the suspends) to the caller. bar()
协程的下一个状态。 Once bar()
returns a value, we can immediately proceed to calling baz()
and returning the value via the StopIteration
exception. bar()
返回一个值,我们就可以立即继续调用baz()
并通过StopIteration
异常返回值。 So the async def coro()
definition shown above can be thought of as syntactic sugar for the following: 因此,上面显示的
async def coro()
定义可以被认为是以下的语法糖:
class coro:
def __init__(self):
self._state = 0
def __iter__(self):
return self
def __await__(self):
return self
def __next__(self):
if self._state == 0:
self._x = foo()
self._bar_iter = bar().__await__()
self._state = 1
if self._state == 1:
try:
suspend_val = next(self._bar_iter)
# propagate the suspended value to the caller
# don't change _state, we will return here for
# as long as bar() keeps suspending
return suspend_val
except StopIteration as stop:
# we got our value
y = stop.value
# since we got the value, immediately proceed to
# invoking `baz`
baz(self._x, y)
self._state = 2
# tell the caller that we're done and inform
# it of the return value
raise StopIteration(42)
# the final state only serves to disable accidental
# resumption of a finished coroutine
raise RuntimeError("cannot reuse already awaited coroutine")
We can test that our "coroutine" works using real asyncio: 我们可以测试我们的“coroutine”使用真正的asyncio:
>>> class coro:
... (definition from above)
...
>>> def foo():
... print('foo')
... return 20
...
>>> async def bar():
... print('bar')
... return 10
...
>>> def baz(x, y):
... print(x, y)
...
>>> asyncio.run(coro())
foo
bar
20 10
42
The remaining part is to write the coro
class in Python/C or in pybind11. 剩下的部分是用Python / C或pybind11编写
coro
类。
For things like this, if I don't want to delve too deep into CPython API, I just write my stuff in Python, and call that using pybind
s Python interface. 对于这样的事情,如果我不想深入研究CPython API,我只需在Python中编写我的东西,并使用
pybind
的Python接口调用它。
An example: https://github.com/RobotLocomotion/drake/blob/a7700d3/bindings/pydrake/ init .py#L44 https://github.com/RobotLocomotion/drake/blob/a7700d3/bindings/pydrake/pydrake_pybind.h#L359 举个例子: https://github.com/RobotLocomotion/drake/blob/a7700d3/bindings/pydrake/ 初始化的.py#L44 https://github.com/RobotLocomotion/drake/blob/a7700d3/bindings/pydrake/pydrake_pybind。 ^ h#L359
Rendering onto this use case, perhaps you can do: 渲染到这个用例,也许你可以这样做:
# cpp_helpers.py
def await_(obj):
return await obj
py::object py_await = py::module::import("cpp_helpers").attr("await_");
auto result = py::cast<MyResult>(py_await(py_obj));
However, this will very likely be less performant than the above solutions. 但是,这很可能不如上述解决方案那么高效。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.