I'm trying to take some functions from a large C++ shared library (libbig.so) and expose them to Python via Cython. To do so, I've got a little C++ file (small.cpp) that provides a thin wrapper around the functionality from the shared library that I need, in a way that makes it easy to call via Cython (pysmall.pyx).
libbig.so -> small.cpp, small.h -> libsmall.so -> pysmall.pyx -> pysmall.cpp -> pysmall.so
I can build and run this extension module on my own computer: I just compile small.cpp into libsmall.so, and then say "libraries=['small']" in the Extension object in setup.py to build the extension module pysmall.so.
I'm now trying to distribute this extension module, and I am having a hard time tracking down resources that describe setup.py best practices for distributing a Cython module as well as C source and shared libraries. I've read through " Installing Python Modules ", " Distributing Python Modules ", and " Distributing Cython Modules ". I understand how to distribute an extension module on its own. I'm less certain about the best way to distribute the dependencies of the extension module.
The Cython documentation indicates that you should include the generated .cpp files as well as the .pyx files, in case Cython is not present, yet it does not provide code to demonstrate how to best handle each situation. It also does not mention how to distribute the shared libraries on which the Cython module depends.
I'm digging through the setup.py scripts from pandas, lxml, pyzmq, h5py, and more, and there's quite a bit of extraneous work happening. If anyone has pointers or example code that might accelerate this process, I'd certainly appreciate it!
1) Distributing libbig.so
This is a problem that python isn't going to help you with. Who are you targeting? If it's linux, can you request that they install it with their package manager? If libbig isn't distributed through a package manager or it's not linux and you're targeting multiple architectures, you might have to distribute the libbig source.
2) Cython/setuptools.
Frankly, I think its easiest to just require that people have Cython. This way there's only one ground truth version of the code, and you don't have to worry about inconsistencies between the .pyx
and .cpp
code. The easiest way to do this is to use setuptools
instead of distutils
. That way, you can use:
setup('mypackage',
...
install_requires=['cython'])
In total, your setup.py
script will look something like:
# setup.py
from setuptools import setup, Extension
from Cython.Distutils import build_ext
pysmall = Extension('pysmall',
sources = ['pysmall.pyx', 'small.cpp'],
include_dirs = ['include/'])
setup(name='mypackage',
packages=['yourpurepythonpackage'],
install_requires=['cython==0.17'],
ext_modules=[pysmall],
cmdclass = {'build_ext': build_ext})
If you don't like the idea of requiring cython, you could do something like:
# setup.py
import warnings
try:
from Cython.Distutils import build_ext
from setuptools import setup, Extension
HAVE_CYTHON = True
except ImportError as e:
HAVE_CYTHON = False
warnings.warn(e.message)
from distutils.core import setup, Extension
from distutils.command import build_ext
pysmall = Extension('pysmall',
sources = ['pysmall.pyx', 'small.cpp'],
include_dirs = ['include/'])
configuration = {'name': 'mypackage',
'packages': ['yourpurepythonpackage'],
'install_requires': ['cython==0.17'],
'ext_modules': [pysmall],
'cmdclass': {'build_ext': build_ext}}
if not HAVE_CYTHON:
pysmall.sources[0] = 'pysmall.cpp'
configuration.pop('install_requires')
setup(**configuration)
Here's my tricky solution. The idea is to "hide" the presence of cython
until it's installed by the requirements. This can be achieved by lazy evaluation. Here's an example:
from setuptools import setup, Extension
class lazy_cythonize(list):
def __init__(self, callback):
self._list, self.callback = None, callback
def c_list(self):
if self._list is None: self._list = self.callback()
return self._list
def __iter__(self):
for e in self.c_list(): yield e
def __getitem__(self, ii): return self.c_list()[ii]
def __len__(self): return len(self.c_list())
def extensions():
from Cython.Build import cythonize
ext = Extension('native_ext_name', ['your/src/*.pyx'])
return cythonize([ext])
configuration = {
'name': 'mypackage',
'packages': ['yourpurepythonpackage'],
'install_requires': ['cython==0.17'],
'ext_modules': lazy_cythonize(extensions)
}
setup(**configuration)
lazy_cythonize is a fake list that generates its internal elements only when someone tries to access to it.
When it's required, this class imports Cython.Build
and generates the list of extensions. This avoids to keep the *.c
files in your project, requiring cython to be installed when the module is building.
Quite tricky, but actually it's working.
I've pushed a fix for setuptools 288 , slated for release as setuptools 18.0. This changelog entry describes a technique that should work with that build. A beta release is available for testing.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.