[英]Wrap Complex C++ Class using the Python Extension API
I'm pretty new to creating C++ class that I can use from within Python.我对创建可以在 Python 中使用的 C++ 类非常陌生。 I've skimmed through a lot of posts on the internet.
我浏览了互联网上的很多帖子。 Be it on StackOverflow, gist, github, ... I've also read the documentation, but I'm not sure how I can solve my issue.
无论是在 StackOverflow、gist、github 上,...我也阅读了文档,但我不确定如何解决我的问题。
Basically, the idea is to do this: http://www.speedupcode.com/c-class-in-python3/ As I want to avoid the burden of creating my own python newtype , I thought that using PyCapsule_New
and PyCapsule_GetPointer
as in the example above could be a workaround, but maybe I'm misleading, and I still need to create complex datatype.基本上,这个想法是这样做的: http : //www.speedupcode.com/c-class-in-python3/因为我想避免创建自己的python
PyCapsule_New
的负担,我认为使用PyCapsule_New
和PyCapsule_GetPointer
就像上面的例子可能是一种解决方法,但也许我在误导,我仍然需要创建复杂的数据类型。
Here is the header of my class I want to be able to call from python:这是我希望能够从 python 调用的类的标题:
template<typename T>
class Graph {
public:
Graph(const vector3D<T>& image, const std::string& similarity, size_t d) : img(image) {...}
component<T> method1(const int k, const bool post_processing=true);
private:
caller_map<T> cmap;
vector3D<T> img; // input image with 3 channels
caller<T> sim; // similarity function
size_t h; // height of the image
size_t w; // width of the image
size_t n_vertices; // number of pixels in the input image
size_t conn; // radius for the number of connected pixels
vector1D<edge<T>> edges; // graph = vector of edges
void create_graph(size_t d);
tuple2 find(vector2D<subset>& subsets, tuple2 i);
void unite(vector2D<subset>& subsets, tuple2 x, tuple2 y);
};
So As you can see my class contains complex structures.所以正如你所看到的,我的班级包含复杂的结构。
vector1D
is just std::vector
but edge is a structure defined by vector1D
只是std::vector
但边缘是由定义的结构
template<typename T>
struct edge {
tuple2 src;
tuple2 dst;
T weight;
};
and some methods use other complex structures.有些方法使用其他复杂的结构。
Anyway, I have created my own Python binding.无论如何,我已经创建了自己的 Python 绑定。 Here I only put the relevant functions.
这里我只放了相关的功能。 I created my
constructor
as follow:我创建了我的
constructor
如下:
static PyObject *construct(PyObject *self, PyObject *args, PyObject *kwargs) {
// Arguments passed from Python
PyArrayObject* arr = nullptr;
// Default if arguments not given
const char* sim = "2000"; // similarity function used
const size_t conn = 1; // Number of neighbor pixels to consider
char *keywords[] = {
"image",
"similarity",
"d",
nullptr
};
if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&|sI:vGraph", keywords, PyArray_Converter, &arr, &sim, &conn)) {
// Will need to DECRF(arr) somewhere?
return nullptr;
}
set<string> sim_strings = {"1976", "1994", "2000"};
if (sim_strings.find(sim) == sim_strings.end()) {
PyErr_SetString(PyExc_ValueError, "This similarity function does not exist");
Py_RETURN_NONE;
}
// Parse the 3D numpy array to vector3D
vector3D<float> img = parse_PyArrayFloat<float>(arr);
// call the Constructor
Graph<float>* graph = new Graph<float>(img, sim, conn);
// Create Python capsule with a pointer to the `Graph` object
PyObject* graphCapsule = PyCapsule_New((void * ) graph, "graphptr", vgraph_destructor);
// int success = PyCapsule_SetPointer(graphCapsule, (void *)graph);
// Return the Python capsule with the pointer to `Graph` object
// return Py_BuildValue("O", graphCapsule);
return graphCapsule;
}
While debugging my code, I can see that my constructor return my graphCapsule object and that it is different from nullptr
.在调试我的代码时,我可以看到我的构造函数返回了我的 graphCapsule 对象,并且它与
nullptr
不同。
then I create my method1
function as follow:然后我创建我的
method1
函数如下:
static PyObject *method1(PyObject *self, PyObject *args) {
// Capsule with the pointer to `Graph` object
PyObject* graphCapsule_;
// Default parameters of the method1 function
size_t k = 300;
bool post_processing = true;
if (!PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)) {
return nullptr;
}
// Get the pointer to `Graph` object
Graph<float>* graph = reinterpret_cast<Graph<float>* >(PyCapsule_GetPointer(graphCapsule_, "graphptr"));
// Call method1
component<float> ctov = graph->method1(k, post_processing);
// Convert component<float> to a Python dict (bad because we need to copy?)
PyObject* result = parse_component<float>(ctov);
return result;
}
When I compile everything, I will have a vgraph.so
library and I will call it from Python using:当我编译所有内容时,我将有一个
vgraph.so
库,我将使用以下命令从 Python 中调用它:
import vgraph
import numpy as np
import scipy.misc
class Vgraph():
def __init__(self, img, similarity, d):
self.graphCapsule = vgraph.construct(img, similarity, d)
def method1(self, k=150, post_processing=True):
vgraph.method1(self.graphCapsule, k, post_processing)
if __name__ == "__main__":
img = scipy.misc.imread("pic.jpg")
img = scipy.misc.imresize(img, (512, 512)) / 255
g = Vgraph(lab_img, "1976", d=1)
cc = g.method1(k=150, post_processing=False)
The idea is that I save the PyObject pointer
returned by the vgraph.construct
.这个想法是我保存
vgraph.construct
返回的PyObject pointer
。 Then I call method1
passing the PyObject pointer
the int k = 150
and the bool postprocessing
.然后我调用
method1
传递PyObject pointer
int k = 150
和bool postprocessing
。
This is why in the C++ implementation of *method1
, I use: !PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)
to parse these 3 objects.这就是为什么在
*method1
的 C++ 实现中,我使用: !PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)
来解析这 3 个对象。
The problem is, even though, when I'm debugging, I recover k=150
and post_processing=False
which come from the way I'm calling the C++ from Python... I'm also getting a 0X0
, that is to say a nullptr
in the variable graphCapsule_
...问题是,即使我在调试时,我恢复了
k=150
和post_processing=False
,它们来自我从 Python 调用 C++ 的方式......我也得到了0X0
,也就是说变量graphCapsule_
的nullptr
...
So obviously the rest of the code cannot work...所以显然其余的代码无法工作......
I thought that PyObject *
is a pointer to my graph of type Graph<float> *
, so, I was expecting ParseTuple to recover my PyObject *
pointer that I can then use in PyCapsule_GetPointer
to retrieve my Object.我认为
PyObject *
是一个指向我的类型的曲线Graph<float> *
,因此,我期待ParseTuple恢复我PyObject *
指针我然后可以在使用PyCapsule_GetPointer
检索我的对象。
How can I make my code work?我怎样才能让我的代码工作? Do I need to define my own PyObject so that ParseTuple understand it?
我是否需要定义我自己的 PyObject 以便 ParseTuple 理解它? Is there a simpler way to do it?
有没有更简单的方法来做到这一点?
Thanks a lot!非常感谢!
Note : If I break in my python code, I can see that my graph g
contains a PyObject
with the address it points to and the name of the object (here graphtr
) so I was expecting my code to work...注意:如果我打破了我的 python 代码,我可以看到我的图形
g
包含一个PyObject
及其指向的地址和对象的名称(这里是graphtr
)所以我希望我的代码能够工作......
Note2 : If I need to create my own newtype
, I have seen this stackoverflow post: How to wrap a C++ object using pure Python Extension API (python3)?注意2 :如果我需要创建自己的
newtype
,我已经看到了这个stackoverflow帖子: How to wrap a C++ object using pure Python Extension API (python3)? but I think because of the complex objects of my Class, it will be quite difficult?但是我觉得因为我的Class对象比较复杂,会不会比较难?
I answer my own question.我回答我自己的问题。
I actually found the flaw in my code.我实际上在我的代码中发现了这个缺陷。
Both functions PyCapsule_GetPointer
and PyCapsule_New
work perfectly fine.这两个函数
PyCapsule_GetPointer
和PyCapsule_New
工作得很好。 As mentioned in my question, the issue arouse just after I was trying to Parse the capsule with the following code:正如我的问题中提到的,在我尝试使用以下代码解析胶囊后,问题就出现了:
size_t k = 300;
bool post_processing = true;
if (!PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)) {
return nullptr;
}
The problem comes from the parsing of the other parameters.问题来自其他参数的解析。 Indeed, k is a
size_t
type so instead of using I
for unsigned int, I should use n
as the documentation mentions:实际上,k 是
size_t
类型,所以我应该使用n
作为文档中提到的,而不是使用I
表示 unsigned int:
n (int) [Py_ssize_t]
Convert a Python integer to a C Py_ssize_t.
Moreover, post_processing
is a boolean so, even though the documentation mentions that a boolean can be parsed with a p
:此外,
post_processing
是一个布尔值,因此,即使文档提到可以使用p
解析布尔值:
p (bool) [int]
I should initialize the boolean with type int
instead of type bool
as it is mentioned in this stackoverflow post我应该用类型
int
而不是类型bool
初始化布尔值,因为它在这个stackoverflow 帖子中提到
So, the working piece of code is:因此,工作代码是:
size_t k = 300;
int post_processing = true;
if (!PyArg_ParseTuple(args, "O|np", &graphCapsule_, &k, &post_processing)) {
return nullptr;
}
We can also use the O!
我们也可以使用
O!
options by passing the &Pycapsule_Type
:通过传递
&Pycapsule_Type
选项:
#include <pycapsule.h>
...
size_t k = 300;
int post_processing = true;
if (!PyArg_ParseTuple(args, "O!|np", &PyCapsule_Type, &graphCapsule_, &k, &post_processing)) {
return nullptr;
}
Finally, as mentioned in my question, It is actually straightforward to implement your own Python type based on this stackoverflow post .最后,正如我的问题中提到的,基于此stackoverflow post实现您自己的 Python 类型实际上很简单。 I have just copied/pasted and adapted the code to my need and it works like a charm without need to use
PyCaspule
anymore!我刚刚复制/粘贴并根据我的需要调整了代码,它就像一个魅力,不再需要使用
PyCaspule
!
Other useful pieces of information :其他有用的信息:
To debug your code (I used vscode on Linux), you can used mixed language debugging.要调试您的代码(我在 Linux 上使用了 vscode),您可以使用混合语言调试。 The idea is to compile your C++ code into as shared library
.so
.这个想法是将您的 C++ 代码编译为共享库
.so
。
Once you're code is compiled you can import it in python:编译代码后,您可以将其导入python:
import my_lib
where my_lib
referred to my_lib.so
file that you have generated.其中
my_lib
指的是您生成的my_lib.so
文件。
To generate your .so
file, you just need to execute: g++ my_python_to_cpp_wrapper.cpp --ggdb -o my_python_to_cpp_wrapper.so
要生成
.so
文件,您只需要执行: g++ my_python_to_cpp_wrapper.cpp --ggdb -o my_python_to_cpp_wrapper.so
But, if you do this, you might missed to include the python library and stuff...但是,如果你这样做,你可能会错过包含 python 库和东西......
Fortunately, python provide a way to find the recommended flags for both compilation and linking :幸运的是,python 提供了一种为编译和链接查找推荐标志的方法:
you just need to execute (change your python version or look into /usr/local/bin eventually)你只需要执行(更改你的python版本或最终查看/usr/local/bin)
/usr/bin/python3.6m-config --cflags
For me, it returned:对我来说,它返回:
-I/usr/include/python3.6m -I/usr/include/python3.6m -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.6-0aiVHW/python3.6-3.6.9=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall
The same applying for linking (change your python version or look into /usr/local/bin eventually)同样申请链接(更改您的python版本或最终查看/usr/local/bin)
/usr/bin/python3.6m-config --ldflags
gave me:给我:
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
Then, since we want to create a shared library .so
we need to add the -shared
flag as well as the -fPIC
flags (otherwise it will complain).然后,因为我们要创建一个共享库
.so
我们需要添加-shared
标志还有-fPIC
标志(否则它会抱怨)。 Finally, since we want to debug our code we should remove any -Ox
such as -O2
or -O3
flag that optimizes the code, because during the debugging you will be prompt with <optimized out>
.最后,因为我们想要调试我们的代码,所以我们应该删除任何
-Ox
比如优化代码的-O2
或-O3
标志,因为在调试过程中,你会得到<optimized out>
提示。 To avoid this, remove any optimization flags from your g++
options.为避免这种情况,请从
g++
选项中删除任何优化标志。 For example, in my case, my cpp file is called: vrgaph.cpp and here is how I compiled it:例如,就我而言,我的cpp文件名为: vrgaph.cpp ,这是我编译它的方式:
g++ vgraph.cpp -ggdb -o vgraph.so -I/usr/include/python3.6m -I/usr/include/python3.6m -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.6-0aiVHW/python3.6-3.6.9=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -Wall -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -shared -fPIC
You can see I'm usin -O1
instead of -O2
or -O3
.你可以看到我全光照
-O1
,而不是-O2
或-O3
。
Once, compile you will have a .so
file that you can import and use in python.一次,编译你将有一个
.so
文件,你可以在 python 中导入和使用。 For my example, I will have vgraph.so
and in my python code, I can do:对于我的示例,我将使用
vgraph.so
并且在我的 python 代码中,我可以执行以下操作:
import vgraph.so
# rest of the call that use you C++ backend code
Then you can easily debug your C++.然后您就可以轻松调试 C++。 There are some posts on the internet that explains how to do that with vs code/gdb/...
互联网上有一些帖子解释了如何使用 vs code/gdb/...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.