How to interface a c++ function that returns a reference to an object via cython

Question

I'm trying to interface a C++ function that returns a reference to an object:

const blob &get_blob();

For cython, i use a .pxd file that gives me access to the C++ namespace:

const blob &get_blob() except +

I then use the function in a .pyx file:

cdef const blob* o = &get_blob()

So i use the address operator and assign to a pointer variable. However, while compiling with cython version 0.23 I get an error:

error: assigning to 'const blob *' from incompatible type '__Pyx_FakeReference *'

This error does not appear if i compile with cython version 0.27. So I consider this a bug in the older cython version.

The essential question: What is the proper way of interfacing a c++ function that returns a reference via cython. I couldn't find any documentation on that.

Answer 1

Your way of handling references is pretty good.

References are a kind of step-children for Cython. I can only speculate about reasons, my try to explain would be:

CPython/Cython are home in C and not C++, and C doesn't know references, which are mostly a syntactic sugar for pointers which cannot be NULL . The references in C++ have an annoying property, you cannot do something like this:

int &x;
x=some_int_var;

but have to initialize the reference at it's creation and cannot change the reference ever again:

int &x=some_int_var;

However, if you take a look into the C/CPP-code produced by Cython, you will see the C90-pattern of declaring all variables at the beginning of a function (thus, it is possible to create a C90-compliant C-code). It would probably mean a lot of work to change this for C++ in order to be able to use references.

So Cython uses a workaround by defining a FakeReference-template class, which wraps a raw-pointer:

template<typename T>
class __Pyx_FakeReference {
  public:
    __Pyx_FakeReference() : ptr(NULL) { }
    __Pyx_FakeReference(const T& ref) : ptr(const_cast<T*>(&ref)) { }
    T *operator->() { return ptr; }
    T *operator&() { return ptr; }
    operator T&() { return *ptr; }
    template<typename U> bool operator ==(U other) { return *ptr == other; }
    template<typename U> bool operator !=(U other) { return *ptr != other; }
  private:
    T *ptr;
};

So for the following silly Cython-code:

%%cython --cplus 
from libcpp.vector cimport vector
cdef set_first(vector[int] & vect):
     cdef int * first=&(vect.at(0))
     first[0]=10

we would get the following generated C-code for the line cdef int * first=&(vect.at(0)) (only important parts):

static PyObject *__pyx_f_5foo_set_first(std::vector<int>  &__pyx_v_vect) {

   # our variable first, not yet initialized (C90 style):
   int *__pyx_v_first; 

   #Fake reference, initialized with NULL  (C90 style):              
   __Pyx_FakeReference<int> __pyx_t_1; 

   #now we use implicit constructor __Pyx_FakeReference(const T& ref), 
   #be aware of const_cast
   #+ (compiler generated, which is Ok because FakeReference 
   #doesn't own the pointer) assignment-operator
   __pyx_t_1 = __pyx_v_vect.at(0);

   #now, cast FakeReference to pointer `first`
   #using operator&
   __pyx_v_first = (&__pyx_t_1);
   ...
 }

The funny and somewhat strange thing is that we can use references in function-signatures, like above set_first(vector[int] & vect) .

This is probably due to the fact, that the passed arguments don't have to be handled by the Cython and are automatically handled correctly on the cpp-code-level.

Last but not least, let's do a fast check, that everything works as expected:

%%cython --cplus 
from libcpp.vector cimport vector
cdef set_first(vector[int] & vect):
     cdef int * first=&(vect.at(0))
     first[0]=10

def create_list(n):
   cdef vector[int] v=range(n)
   set_first(v)
   return v

>>> create_list(2)
[10,1] # yep it worked!

---

A warning: one might be tempted to try something like:

%%cython --cplus 
from libcpp.vector cimport vector
cdef set_first(vector[int] & vect):
     first=vect.at(0)
     (&first)[0]=10

in hope, that first is somehow magically a (python?) reference. In reality, first is of type int and the last line is a complicated way to set this local variable to 10 and not a way to set the first element in vector. Here are the crucial differences to the version above:

static PyObject *__pyx_f_5foo_set_first(std::vector<int>  &__pyx_v_vect) {

   # "int" not "int *"!
   int __pyx_v_first; 


   #now, cast FakeReference  __pyx_t_1 to int via operator() 
   #which return "*ptr" (and not "ptr" as operator&())
   __pyx_v_first = __pyx_t_1;
   ...
 }