Using data in Python code from C++ using Cython

Question

I'm trying to get list from C++ code in my Python code using Cython.

Here my header (ex.h):

namespace c                   
{                             
class C {                     
    int id;                   
public:                       
    C(int id) {               
        this->id = id;        
    }                         
    int getId();              
};                            
typedef std::vector<C> CList; 
}       

and my source file (ex.cpp):

namespace c                      
{                                
int C::getId() {                 
    return this->id;             
}                                
CList getlist(int t) {           
    CList list;                  
    for (int i = 0; i < t; ++i) {
        C c(i);                  
        list.push_back(c);       
    }                            
    return list;                 
}                                
}                                

So in my Python code I want to iterate through CList like this:

import exapi               

for i in exapi.take_list():
    print i               
    # OR
    print i.get_id()

In .pyx file (exapi.pyx) I'm trying to create Cython extension to handle C++ object:

cdef extern from "ex.h" namespace "c":        
    cdef cppclass C:                          
        C(int) except +                       
        int getId()                           

    cdef cppclass CList:                      
        pass                                  

    cdef CList getlist(int)                   

cdef class PY_C:                              
    pass                                      

cdef class PY_CList:                          
    pass                                      


def take_list():                              
    return getlist(10) # I know this is wrong 

What forwarding methods and function should I add to PY_C and PY_Clist to make my python code working?

And I read Using C++ in Cython , but there new objects creates in Cython code, but in my case I need to use objects created in C++ side.

Answer 1

When trying to work this out I rewrote some of the example, but in principle there's no difference between what you gave and what I'm giving back.

I changed some names, too, because the example ones were too short for me ;).

cobject.hpp

#include <vector>

namespace cobject {
    class CObject {
        int id;
    public:
        CObject(int id): id(id) {}
        int getId();
    };

    typedef std::vector<CObject> CObjectList;
    CObjectList getlist(int t);
}

cobject.cpp

#include "cobject.hpp"

namespace cobject {
    int CObject::getId() {
        return this->id;
    }

    CObjectList getlist(int t) {
        CObjectList list;

        for (int i = 0; i < t; ++i) {
            list.push_back(CObject(i));
        }

        return list;
    }
}

These aren't really any different.

I wanted to use pyximport , but to use it with C++ you need a special file:

cobject_api.pyxbld

import os
from distutils.extension import Extension

dirname = os.path.dirname(__file__)

def make_ext(modname, pyxfilename):
    return Extension(
        name=modname,
        sources=[pyxfilename, "cobject.cpp"],
        language="c++",
        include_dirs=[dirname]
    )

And then the magic:

cython_api.pyx

from cython.operator cimport dereference as deref

cdef extern from "cobject.hpp" namespace "cobject":
    cdef cppclass CObject:
        CObject(int) except +
        int getId()

    cdef cppclass CObjectList:
        CObject &at(size_t)
        size_t size()

    cdef CObjectList getlist(int)


# Cython wrappers

cdef class PyCObject:
    cdef CObject *wrapped
    cdef object keepalive

    def __cinit__(self, keepalive):
        self.keepalive = keepalive

    property id:
        def __get__(self):
            return self.wrapped.getId()

    def __repr__(self):
        return "PyCObject({})".format(self.id)

cdef class PyCObjectList:
    cdef CObjectList wrapped

    def __iter__(self):
        cdef PyCObject pycobject

        cdef size_t idx = 0

        while idx < self.wrapped.size():
            pycobject = PyCObject(self)
            pycobject.wrapped = &self.wrapped.at(idx)

            yield pycobject

            idx += 1

def take_list():
    cdef PyCObjectList pycobjectlist = PyCObjectList()
    pycobjectlist.wrapped = getlist(10)

    return pycobjectlist

Which is run like:

main.py

import pyximport
pyximport.install()

import cobject_api

for i in cobject_api.take_list():
    print("{} has id {}".format(i, i.id))

---

The idea is that your wrapper classes take ownership of the data. Since getlist was returning by value you have to copy it around, but the other object wrapper can use a pointer.

Rather than using a CObjectList type, it might make more sense to use the pre-wrapped vector class and iterate over that. However, unless you do keep everything on the heap with pointers for indirection it's not going to be nice to wrap them.

This is because you can't easily point to things that you keep on the stack, and Cython requires pointing to things if it wants to tie in with Python's garbage collector. This explains what the keepalive variable in the code is for: unlike in Python where deleting a vector doesn't affect its elements, in C++ that trashes them completely. With a vector of pointers, these problems can be alleviated.

Also see this link on another question. Some of the same points have been made, and the architechture for a more traditional setup is given.