Cython: Calling Python code from C program

Question

I am trying to make a Cython wrapper so I can call Python code from C. I am having issues with import as I would like the wrapper to be separate from original code.

Code below ends in segfault when calling imported function. If the code is written as a python module and imported via import the program says that the name ... is not defined . The problem does not exhibit itself when everything is in one file and there's no import involved (indeed code generated by Cython fails when cimporting). The code works fine as well when libcimpy.pyx is imported from other python script (either compiled to .so or live)

I have prepared a minimal example. This is far from actual code but covers the principle.

cimpPy.pyx: Sample python code (converted to Cython)

cdef sum(a, b):
    return a + b

cimpPy.pxd

cdef sum(a, b)

libcimpy.pyx (glue Cython code)

cimport cimpPy

cdef public int cSum(int a, int b):
    return cimpPy.sum(a, b)

ci.c (c code from which we want to call cimpPy)

#include <stdio.h>
#include <stdlib.h>
#include <Python.h>
#include "libcimp.h"

int main(int argc, char **argv) {
  Py_Initialize();
  initlibcimp();
  int a = 2;
  int b = 3;
  int c = cSum(a, b);
  printf("sum of %d and %d is %d\n", a, b, c);
  Py_Finalize();
  return 0;
}

Makefile

EXECUTABLE = ci

OBJS       = ci.o

CC         = gcc
CFLAGS     = -g -I/usr/include/python2.7 -I$(shell pwd)

LINKER     = g++
LDFLAGS    = -L$(shell pwd) $(shell python-config --ldflags) -lcimp
.PHONY: clean cython

all: cython $(EXECUTABLE)

cython:
    python setup.py build_ext --inplace

.c.o:
    $(CC) $(CFLAGS) -c $<

$(EXECUTABLE) : $(OBJS)
    $(LINKER) -o $(EXECUTABLE) $(OBJS) $(LDFLAGS)

clean:
    rm -rf *.o *.so libcimp.c libcimp.h core build $(EXECUTABLE)

setup.py

from distutils.core import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

extensions = [
    Extension("libcimp", ["libcimp.pyx"])
]

setup(
    name = "CIMP",
    cmdclass = {"build_ext": build_ext},
    ext_modules = cythonize(extensions)
)

What I intend to achieve is being able to plug Python code into larger C system. The assumption is that users will be able to write Python themselves. The C code is a simulation engine which can operate on agents in a environment. The idea is that the behaviour of agents and environment can be specified in python and passed to the engine for evaluation when necessary. The best analogy would be a map reduce system where Python scripts are mappers. In this sense I want to call Python from C and not the other way round.

Converting everything to Cython, while compelling would be to large undertaking.

Is this the right approach? Why import works only under python interpreter and not when embedded externally? Any suggestions and reference articles or documentation are appreciated.

Answer 1

In this code, the initlibcimp() actually fails, but you don't see it right away because the error is reported by setting a python exception. I'm not 100% sure this is the correct way to do this, but I could see the error by adding the following code below that call:

if (PyErr_Occurred())
{
    PyErr_Print();
    exit(-1);
}

Then, the program will output:

Traceback (most recent call last):
  File "libcimpy.pyx", line 1, in init libcimpy (libcimpy.c:814)
    cimport cimpPy
ImportError: No module named cimpPy

The reason that the cimpPy module is not yet defined, is that you need to do a call to initcimpPy() before calling initlibcimp .