Calling complicated C++ functions in Python (linux)

Question

I have some functions in C++ source, declared like this:

extern "C" {
    int32_t comp_ip_long(const std::vector<EachColumn>& in, std::vector<EachColumn>& out);
}

The C++ code is compiled into xxx.so (linux shared object) and I want to call the function in Python. notice that I cannot modify the C++ code, so something like boost::python is not accessable.

I've tried ctypes.CDLL, but I don't know how to pass the complicated argument to xxx.so?

PS:

I've given a .so with some functions (with the above parameter type, but function names are unknown), the function name and parameters are input by user.

Answer 1

I believe that you need helper glue functions declared as extern "C" to construct (ie initialize) and fill your std::vector<EachColumn> and have these helper functions called from Python.

Perhaps

typedef std::vector<EachColumn> columnvect_t;
extern "C" columnvect_t *new_vect() 
    { return new columnvect_t; };
extern "C" void del_vect(columnvect_t*vec) 
   { delete vec; };
extern "C" void pushback_vect(columnvect_t* vec, EachColumn* col) 
   { vec->push_back(*col); };

and likewise for your EachColumn class.

^{Maybe you need to make a glue library linked to the C++ library and dlopen -ed by Python}

Basically, Python is more C friendly than C++ friendly (so you need to make C++ glue code feeling like C for Python). Be careful that no C++ exception throwing crosses the Python interpreter (so catch all of them inside your glue functions)

If the C or C++ code is huge, you might consider customizing GCC to perhaps help in generating such glue code, either with MELT or with D.Malcom's GCC python Plugin . But this takes time.

Answer 2

You can't call it from ctypes, you need at least to warp the function inside C so you could call it from Python.

I don't know the details of your function but for example if you have such this C++ code:

#include <iostream>

class Foo {
  int bar;
public:
  Foo(int bar) : bar(bar) {}

  int getBar() const {
    return bar;
  }

  void setBar(int bar) {
    this->bar = bar;
  }

  void doSomething() const {
    std::cout << bar << std::endl;
  }
};

you can warp int this way:

// previous code here +
#define CAST_FOO(x) (static_cast<Foo *>(x))

#ifndef FOO_DEBUG
extern "C" {
#endif

void* foo_new(int bar) {
  return static_cast<void*>(new Foo(bar));
}

int foo_get_bar(void *foo) {
  return CAST_FOO(foo)->getBar();
}

void foo_set_bar(void *foo, int bar) {
  CAST_FOO(foo)->setBar(bar);
}
void foo_do_something(void* foo) {
  CAST_FOO(foo)->doSomething();
}

void foo_destroy(void* foo) {
  delete CAST_FOO(foo);
}

#ifndef FOO_DEBUG
};
#endif

#ifdef FOO_DEBUG
int main(int argc, char **argv) {
  void* foo = foo_new(10);

  foo_do_something(foo);
  foo_set_bar(foo, 20);
  foo_do_something(foo);

  foo_destroy(foo);

  return 0;
}
#endif

now it should be callable from ctypes, and from C also.

$ g++ -Wall foo.cpp -DFOO_DEBUG
$ ./a.out
10
20
$ g++ -Wall foo.cpp -shared -o foo.so
$ python
>>> from ctypes import *
>>>
>>> so = cdll.LoadLibrary('foo.so')
>>> foo = so.foo_new(10)
>>>
>>> so.foo_do_something(foo)
10
>>> so.foo_set_bar(foo, 20)
>>> so.foo_do_something(foo)
20
>>> so.foo_destroy(foo)
>>>

Answer 3

A bit more work but maybe worth it in the long run is to use Boost.Python

Example taken from the boost documentation:

Following C/C++ tradition, let's start with the "hello, world". A C++ Function:

char const* greet()
{
    return "hello, world";
}

can be exposed to Python by writing a Boost.Python wrapper:

#include <boost/python.hpp>

BOOST_PYTHON_MODULE(hello_ext)
{
    using namespace boost::python;
    def("greet", greet);
}

That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python. Here's a sample Python session:

>>> import hello_ext
>>> print hello_ext.greet()
hello, world