Passing complex linking flags from Makefile to setup.py

Question

I am writing Python 2.7 C++ extension. The Python extension wraps part of C++ library (Kaldi). Previously I created shared library and distributed it with the Python extension. (I needed set up LD_LIBRARY_PATH).

I want to compile the extension statically. The Kaldi library can be compiled statically (with -fPIC flag). The problem is that the compilation depends on other libraries and flags are generated by the configure script.

I want to compile the extension using setup.py and "steal" the compilation setup from Makefile . How would you do it?

The command for linking the shared library was:

$(CXX) -shared -DPIC -o $@ -Wl,-soname=$@,--whole-archive $^ -Wl,--no-whole-archive $(EXTRA_LDLIBS) $(LDFLAGS) $(LDLIBS)

In the setup.py I had:

ext_modules.append(Extension('pykaldi.decoders',
                         language='c++',
                         include_dirs=['..', 'fst'],
                         library_dirs=['.'],
                         libraries=['pykaldi'],
                         sources=['pykaldi/decoders.pyx'],
                         ))

The $(EXTRA_LDLIBS) , $(LDFLAGS) and $(LDLIBS) are generated by configure script. The $(LDLIBS) contains some static libraries some shared.

NOW I have

if STATIC:
    # STATIC 
    # TODO extract linking parameters from Makefile
    library_dirs, libraries = [], []
    extra_objects = ['pykaldi.a', ]
else:
    # DYNAMIC
    library_dirs = ['.'],
    libraries = ['pykaldi']
    extra_objects = []
ext_modules.append(Extension('pykaldi.decoders',
                             language='c++',
                             include_dirs=['..', 'fst'],
                             library_dirs=library_dirs,
                             libraries=libraries,
                             extra_objects=extra_objects,
                             sources=['pykaldi/decoders.pyx'],
                             ))

Note 1: I am using Cython, but it should not matter.

Note 2: I know I can compile the extension using make , I would prefer setup.py for better deployment.

Answer 1

You would have to link a static extension to python when you build the python executable.

That is because extensions are loaded into the interpreter using dlopen (or LoadLibrary on Windows) and this functionality can not support static 'libraries' which are nothing more than a tarball of the object files. Object files are a raw form still unusable by the system runtime until they've gone through a linking phase.