Wrap Complex C++ Class using the Python Extension API

Question

I'm pretty new to creating C++ class that I can use from within Python. I've skimmed through a lot of posts on the internet. Be it on StackOverflow, gist, github, ... I've also read the documentation, but I'm not sure how I can solve my issue.

Basically, the idea is to do this: http://www.speedupcode.com/c-class-in-python3/ As I want to avoid the burden of creating my own python newtype , I thought that using PyCapsule_New and PyCapsule_GetPointer as in the example above could be a workaround, but maybe I'm misleading, and I still need to create complex datatype.

Here is the header of my class I want to be able to call from python:

template<typename T>
class Graph {
    public:
        Graph(const vector3D<T>& image, const std::string& similarity, size_t d) : img(image) {...}
        component<T> method1(const int k, const bool post_processing=true);

    private:
        caller_map<T> cmap;
        vector3D<T> img;  // input image with 3 channels
        caller<T> sim;  // similarity function
        size_t h;  // height of the image
        size_t w;  // width of the image
        size_t n_vertices;  // number of pixels in the input image
        size_t conn;  // radius for the number of connected pixels
        vector1D<edge<T>> edges;  // graph = vector of edges

        void create_graph(size_t d);
        tuple2 find(vector2D<subset>& subsets, tuple2 i);
        void unite(vector2D<subset>& subsets, tuple2 x, tuple2 y);
};

So As you can see my class contains complex structures. vector1D is just std::vector but edge is a structure defined by

template<typename T>
struct edge {
    tuple2 src;
    tuple2 dst;
    T weight;
};

and some methods use other complex structures.

Anyway, I have created my own Python binding. Here I only put the relevant functions. I created my constructor as follow:

static PyObject *construct(PyObject *self, PyObject *args, PyObject *kwargs) {
    // Arguments passed from Python
    PyArrayObject* arr = nullptr;

    // Default if arguments not given
    const char* sim = "2000";   // similarity function used
    const size_t conn = 1;  // Number of neighbor pixels to consider

    char *keywords[] = {
        "image",
        "similarity",
        "d",
        nullptr
    };

    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&|sI:vGraph", keywords, PyArray_Converter, &arr, &sim, &conn)) {
        // Will need to DECRF(arr) somewhere?
        return nullptr;
    }

    set<string> sim_strings = {"1976", "1994", "2000"};

    if (sim_strings.find(sim) == sim_strings.end()) {
        PyErr_SetString(PyExc_ValueError, "This similarity function does not exist");
        Py_RETURN_NONE;
    }

    // Parse the 3D numpy array to vector3D
    vector3D<float> img = parse_PyArrayFloat<float>(arr);

    // call the Constructor
    Graph<float>* graph = new Graph<float>(img, sim, conn);

    // Create Python capsule with a pointer to the `Graph` object
    PyObject* graphCapsule = PyCapsule_New((void * ) graph, "graphptr", vgraph_destructor);

    // int success = PyCapsule_SetPointer(graphCapsule, (void *)graph);
    // Return the Python capsule with the pointer to `Graph` object
    // return Py_BuildValue("O", graphCapsule);
    return graphCapsule;
}

While debugging my code, I can see that my constructor return my graphCapsule object and that it is different from nullptr .

then I create my method1 function as follow:

static PyObject *method1(PyObject *self, PyObject *args) {
    // Capsule with the pointer to `Graph` object
    PyObject* graphCapsule_;

    // Default parameters of the method1 function
    size_t k = 300;
    bool post_processing = true;

    if (!PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)) {
        return nullptr;
    }

    // Get the pointer to `Graph` object
    Graph<float>* graph = reinterpret_cast<Graph<float>* >(PyCapsule_GetPointer(graphCapsule_, "graphptr"));

    // Call method1
    component<float> ctov = graph->method1(k, post_processing);

    // Convert component<float> to a Python dict (bad because we need to copy?)
    PyObject* result = parse_component<float>(ctov);

    return result;
}

When I compile everything, I will have a vgraph.so library and I will call it from Python using:

import vgraph
import numpy as np
import scipy.misc

class Vgraph():
    def __init__(self, img, similarity, d):
        self.graphCapsule = vgraph.construct(img, similarity, d)

    def method1(self, k=150, post_processing=True):
        vgraph.method1(self.graphCapsule, k, post_processing)

if __name__ == "__main__":
    img = scipy.misc.imread("pic.jpg")
    img = scipy.misc.imresize(img, (512, 512)) / 255

    g = Vgraph(lab_img, "1976", d=1)
    cc = g.method1(k=150, post_processing=False)

The idea is that I save the PyObject pointer returned by the vgraph.construct . Then I call method1 passing the PyObject pointer the int k = 150 and the bool postprocessing .

This is why in the C++ implementation of *method1 , I use: !PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing) to parse these 3 objects.

The problem is, even though, when I'm debugging, I recover k=150 and post_processing=False which come from the way I'm calling the C++ from Python... I'm also getting a 0X0 , that is to say a nullptr in the variable graphCapsule_ ...

So obviously the rest of the code cannot work...

I thought that PyObject * is a pointer to my graph of type Graph<float> * , so, I was expecting ParseTuple to recover my PyObject * pointer that I can then use in PyCapsule_GetPointer to retrieve my Object.

How can I make my code work? Do I need to define my own PyObject so that ParseTuple understand it? Is there a simpler way to do it?

Thanks a lot!

Note : If I break in my python code, I can see that my graph g contains a PyObject with the address it points to and the name of the object (here graphtr ) so I was expecting my code to work...

Note2 : If I need to create my own newtype , I have seen this stackoverflow post: How to wrap a C++ object using pure Python Extension API (python3)? but I think because of the complex objects of my Class, it will be quite difficult?

Answer 1

I answer my own question.

I actually found the flaw in my code.

Both functions PyCapsule_GetPointer and PyCapsule_New work perfectly fine. As mentioned in my question, the issue arouse just after I was trying to Parse the capsule with the following code:

size_t k = 300;
bool post_processing = true;

if (!PyArg_ParseTuple(args, "O|Ip", &graphCapsule_, &k, &post_processing)) {
    return nullptr;
}

The problem comes from the parsing of the other parameters. Indeed, k is a size_t type so instead of using I for unsigned int, I should use n as the documentation mentions:

n (int) [Py_ssize_t]
Convert a Python integer to a C Py_ssize_t.

Moreover, post_processing is a boolean so, even though the documentation mentions that a boolean can be parsed with a p :

p (bool) [int]

I should initialize the boolean with type int instead of type bool as it is mentioned in this stackoverflow post

So, the working piece of code is:

size_t k = 300;
int post_processing = true;

if (!PyArg_ParseTuple(args, "O|np", &graphCapsule_, &k, &post_processing)) {
    return nullptr;
}

We can also use the O! options by passing the &Pycapsule_Type :

#include <pycapsule.h>
...
size_t k = 300;
int post_processing = true;

if (!PyArg_ParseTuple(args, "O!|np", &PyCapsule_Type, &graphCapsule_, &k, &post_processing)) {
    return nullptr;
}

Finally, as mentioned in my question, It is actually straightforward to implement your own Python type based on this stackoverflow post . I have just copied/pasted and adapted the code to my need and it works like a charm without need to use PyCaspule anymore!

Other useful pieces of information :

To debug your code (I used vscode on Linux), you can used mixed language debugging. The idea is to compile your C++ code into as shared library .so .

Once you're code is compiled you can import it in python:

import my_lib

where my_lib referred to my_lib.so file that you have generated.

To generate your .so file, you just need to execute: g++ my_python_to_cpp_wrapper.cpp --ggdb -o my_python_to_cpp_wrapper.so

But, if you do this, you might missed to include the python library and stuff...

Fortunately, python provide a way to find the recommended flags for both compilation and linking :

you just need to execute (change your python version or look into /usr/local/bin eventually)

/usr/bin/python3.6m-config --cflags

For me, it returned:

-I/usr/include/python3.6m -I/usr/include/python3.6m  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.6-0aiVHW/python3.6-3.6.9=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall

The same applying for linking (change your python version or look into /usr/local/bin eventually)

/usr/bin/python3.6m-config --ldflags

gave me:

-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

Then, since we want to create a shared library .so we need to add the -shared flag as well as the -fPIC flags (otherwise it will complain). Finally, since we want to debug our code we should remove any -Ox such as -O2 or -O3 flag that optimizes the code, because during the debugging you will be prompt with <optimized out> . To avoid this, remove any optimization flags from your g++ options. For example, in my case, my cpp file is called: vrgaph.cpp and here is how I compiled it:

g++ vgraph.cpp -ggdb -o vgraph.so -I/usr/include/python3.6m -I/usr/include/python3.6m  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.6-0aiVHW/python3.6-3.6.9=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -Wall -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -shared -fPIC

You can see I'm usin -O1 instead of -O2 or -O3 .

Once, compile you will have a .so file that you can import and use in python. For my example, I will have vgraph.so and in my python code, I can do:

import vgraph.so

# rest of the call that use you C++ backend code

Then you can easily debug your C++. There are some posts on the internet that explains how to do that with vs code/gdb/...