[英]How to invoke Python function as a callback inside C++ thread using pybind11
I designed a C++ system that invokes user defined callbacks from procedure running in a separate thread.我设计了一个 C++ 系统,它从在单独线程中运行的过程调用用户定义的回调。 Simplified
system.hpp
looks like this:简化后的
system.hpp
如下所示:
#pragma once
#include <atomic>
#include <chrono>
#include <functional>
#include <thread>
class System
{
public:
using Callback = std::function<void(int)>;
System(): t_(), cb_(), stop_(true) {}
~System()
{
stop();
}
bool start()
{
if (t_.joinable()) return false;
stop_ = false;
t_ = std::thread([this]()
{
while (!stop_)
{
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if (cb_) cb_(1234);
}
});
return true;
}
bool stop()
{
if (!t_.joinable()) return false;
stop_ = true;
t_.join();
return true;
}
bool registerCallback(Callback cb)
{
if (t_.joinable()) return false;
cb_ = cb;
return true;
}
private:
std::thread t_;
Callback cb_;
std::atomic_bool stop_;
};
It works just fine and can be tested with this short example main.cpp
:它工作得很好,可以用这个简短的例子
main.cpp
进行测试:
#include <iostream>
#include "system.hpp"
int g_counter = 0;
void foo(int i)
{
std::cout << i << std::endl;
g_counter++;
}
int main()
{
System s;
s.registerCallback(foo);
s.start();
while (g_counter < 3)
{
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
s.stop();
return 0;
}
which will output 1234
a few times and then will stop.这将输出
1234
几次,然后将停止。 However I encountered a problem trying to create python bindings for my System
.但是,我在尝试为我的
System
创建 python 绑定时遇到了问题。 If I register a python function as a callback, my program will deadlock after calling System::stop
.如果我注册一个 python 函数作为回调,我的程序将在调用
System::stop
后死锁。 I investigated the topic a bit and it seems that I face the issue with GIL .我调查了这个话题,似乎我面临着GIL的问题。 Reproducible example:
可重现的例子:
binding.cpp
: binding.cpp
:
#include "pybind11/functional.h"
#include "pybind11/pybind11.h"
#include "system.hpp"
namespace py = pybind11;
PYBIND11_MODULE(mysystembinding, m) {
py::class_<System>(m, "System")
.def(py::init<>())
.def("start", &System::start)
.def("stop", &System::stop)
.def("registerCallback", &System::registerCallback);
}
python script:蟒蛇脚本:
#!/usr/bin/env python
import mysystembinding
import time
g_counter = 0
def foo(i):
global g_counter
print(i)
g_counter = g_counter + 1
s = mysystembinding.System()
s.registerCallback(foo)
s.start()
while g_counter < 3:
time.sleep(1)
s.stop()
I have read the pybind11 docs section about the possibility to acquire or release GIL on the C++ side.我已阅读有关在 C++ 端获取或释放 GIL 的可能性的pybind11 文档部分。 However I did not manage to get rid of the deadlock that occurs in my case:
但是我没有设法摆脱在我的情况下发生的僵局:
PYBIND11_MODULE(mysystembinding, m) {
py::class_<System>(m, "System")
.def(py::init<>())
.def("start", &System::start)
.def("stop", &System::stop)
.def("registerCallback", [](System* s, System::Callback cb)
{
s->registerCallback([cb](int i)
{
// py::gil_scoped_acquire acquire;
// py::gil_scoped_release release;
cb(i);
});
});
}
If I call py::gil_scoped_acquire acquire;
如果我调用
py::gil_scoped_acquire acquire;
before calling the callback, deadlock occurs anyway.在调用回调之前,无论如何都会发生死锁。 If I call
py::gil_scoped_release release;
如果我调用
py::gil_scoped_release release;
before calling the callback, I get在调用回调之前,我得到
Fatal Python error: PyEval_SaveThread: NULL tstate
致命的 Python 错误:PyEval_SaveThread:NULL tstate
What should I do to register python functions as callbacks and avoid deadlocks?我应该怎么做才能将 python 函数注册为回调并避免死锁?
Thanks to this discussion and many other resources ( 1 , 2 , 3 ) I figured out that guarding the functions that start and join the C++ thread with gil_scoped_release
seems to solve the problem:由于这个讨论和许多其他资源( 1 、 2 、 3 ),我发现用
gil_scoped_release
保护启动和加入 C++ 线程的gil_scoped_release
似乎可以解决问题:
PYBIND11_MODULE(mysystembinding, m) {
py::class_<System>(m, "System")
.def(py::init<>())
.def("start", &System::start, py::call_guard<py::gil_scoped_release>())
.def("stop", &System::stop, py::call_guard<py::gil_scoped_release>())
.def("registerCallback", &System::registerCallback);
}
Apparently deadlocks occurred because python was holding a lock while invoking a binding responsible for C++ thread manipulation.显然,死锁的发生是因为 python 在调用负责 C++ 线程操作的绑定时持有锁。 I am still not really sure if my reasoning is correct, so I would appreciate any expert's comments.
我仍然不确定我的推理是否正确,因此我将不胜感激任何专家的评论。
Call gil_scoped_release
before join()
will get rid of the deadlock in my case.在
join()
之前调用gil_scoped_release
将摆脱我的情况下的僵局。
void Tick::WaitLifeOver() {
if (thread_.joinable()) {
thread_.join();
}
}
PYBIND11_MODULE(tick_pb, m) {
py::class_<Tick, std::shared_ptr<Tick>>(m, "Tick")
// ...
.def("wait_life_over", &Tick::WaitLifeOver,
py::call_guard<py::gil_scoped_release>());
}
Here are the codes: C++ Thread Callback Python Function以下是代码: C++ 线程回调 Python 函数
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.