Including a somewhat large C library in Cython

Question

I'm very new to Cython, and know the basics of C/C++ compilation.

Here's what I'm trying to do:

There is a decently large library written in C that I want to incorporate in my Python program. (In particular, it's the implementation of board logic for GNUGo. I'm experimenting with building AIs for it.) I need this routine to be fast, and using a Python implementation of the logic has been too slow.

The Cython documentation says that I should create a .pxd file that lists method signatures that are defined extern from the .h files. I should then create a wrapper class in the corresponding .pyx file, and manually create Python-side function calls that hook into the method signatures listed above.

The issue I have is that the library I want to implement isn't a single C file like the documentation example. It uses multiple C files, with their own interdependencies, and the game board is represented as a set of global variables that are defined extern all over the place. I really, really don't want to write .pxd s for all the required C files.

The GNUGo documentation suggests that I can create a libboard.a file - is this conceptually different from a .o file, and is there a way to include this file into Cython? I ideally want to treat it as a blackbox that lets me do a method call, have the library deal with its crazy global variable nuttiness, and return the desired output.

Answer 1

I'm not sure if this is the "proper" way to do it, but here's how I got it working.

In my go.pxd file, I list all method declarations I need from the header file board.h . Instead of doing

cdef extern from "board.h":
    ...

I do

cdef extern:
    ...

since I don't want to actually include the board.h file - I have the library libboard.a instead. Inside that block I list all the methods I need.

In my go.pyx file, I do cimport go , then do go.method_name(args) whenever I need to use it. Running Cython on go.pyx to generate go.c works file because all the methods are defined extern .

Finally, to turn it into a go.so library to import, I have a Makefile that basically does

gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing \
  -I/usr/include/python2.7 go.c libboard.a -o go.so

And the final go.so is importable in Python and works fine.

The tricky part was that I needed to recompile libboard.a with -fPIC before linking it. I also needed to remove some library calls to curses.h/term.h since I wasn't able to get those includes to work, but luckily I didn't need the pretty-printing anyways.

In general, it seems like you should be able to link arbitrary .o or .a declarations in the final step, as long as you don't specify where they are extern from in the .pxd file. That seems like it could get very messy very fast, but at least it works.