How to distribute a module with both a pure Python and Cython version

Question

I have a pure Python module and I want to rewrite some of submodules using Cython. Then I would like to add the new Cython submodules to the original Python module and make them available only as an option, meaning that cythoning the module is not compulsory (in which case the 'old' pure Python module should be used).

Here is an example:

my_module
    - __init__.py
    - a.py
    - b.py
    - setup.py

where a.py contains import b .

I want to write b.py in Cython. The idea would be to add a folder containing the .pyx file, for example:

my_module
    - __init_.py
    - a.py
    - b.py
    - setup.py
    cython
        -b.pyx

setup.py would contain the direction to compile b.pyx and to install the module. However, I would like that if someone runs python setup.py install then the pure Python code is installed, whereas if an option is added then the Cython code is compiled and installed.

Any idea how to do that?

Also, how should the file a.py be modified in order to import the correct module?

Answer 1

I am not sure about your setup.py requirement (I don't know why you would need that) but as for the runtime import issue, I wrote a decorator to do just that:

from __future__ import print_function
from importlib import import_module
from functools import wraps
import inspect
import sys

MAKE_NOISE = False

def external(f):
    """ Decorator that looks for an external version of
        the decorated function -- if one is found and
        imported, it replaces the decorated function
        in-place (and thus transparently, to would-be
        users of the code). """
    f.__external__ = 0 # Mark func as non-native

    function_name = hasattr(f, 'func_name') and f.func_name or f.__name__
    module_name = inspect.getmodule(f).__name__

    # Always return the straight decoratee func,
    # whenever something goes awry. 
    if not function_name or not module_name:
        MAKE_NOISE and print("Bad function or module name (respectively, %s and %s)" % (
            function_name, module_name), file=sys.stderr)
        return f

    # This function is `pylire.process.external()`.
    # It is used to decorate functions in `pylire.process.*`,
    # each of which possibly has a native (Cython) accelerated
    # version waiting to be imported in `pylire.process.ext.*`
    # … for example: if in `pylire/process/my_module.py` you did this:
    # 
    #   @external
    #   def my_function(*args, **kwargs):
    #       """ The slow, pure-Python implementation """
    #       pass
    # 
    # … and you had a Cython version of `my_function()` set up
    # in `pylire/process/ext/my_module.pyx` – you would get the fast
    # function version, automatically at runtime, without changing code.
    #
    # TL,DR: you'll want to change the `pylire.process.ext` string (below)
    # to match whatever your packages' structure looks like.
    module_file_name = module_name.split('.')[-1]
    module_name = "pylire.process.ext.%s" % module_file_name

    # Import the 'ext' version of process
    try:
        module = import_module(module_name)
    except ImportError:
        MAKE_NOISE and print("Error importing module (%s)" % (
            module_name,), file=sys.stderr)
        return f
    MAKE_NOISE and print("Using ext module: %s" % (
        module_name,), file=sys.stderr)

    # Get the external function with a name that
    # matches that of the decoratee.
    try:
        ext_function = getattr(module, function_name)
    except AttributeError:
        # no matching function in the ext module
        MAKE_NOISE and print("Ext function not found with name (%s)" % (
            function_name,), file=sys.stderr)
        return f
    except TypeError:
        # function_name was probably shit
        MAKE_NOISE and print("Bad name given for ext_function lookup (%s)" % (
            function_name,), file=sys.stderr)
        return f

    # Try to set telltale/convenience attributes
    # on the new external function -- this doesn't
    # always work, for more heavily encythoned
    # and cdef'd function examples.
    try:
        setattr(ext_function, '__external__', 1)
        setattr(ext_function, 'orig', f)
    except AttributeError:
        MAKE_NOISE and print("Bailing, failed setting ext_function attributes (%s)" % (
            function_name,), file=sys.stderr)
        return ext_function
    return wraps(f)(ext_function)

… this lets you decorate functions as @external – and they are replaced at runtime automatically with the Cython-optimized versions you've provided.

If you wanted to extend this idea to replacing entire Cythonized classes, it'd be straightforward to use the same logic in the __new__ method of a metaclass (eg opportunistic find-and-replace in the optimized module).

Answer 2

My solution was to set up the module like this:

my_module
    - __init_.py
    - a.py
    - b.py
    - setup.py
    cython_my_module
        - __init_.py
        - b.pyx

The setup.py would contain something similar to this:

from distutils.core import setup
from Cython.Build import cythonize

import numpy

setup(
    name='My_module',
    ext_modules=cythonize(["cython_my_module/b.pyx",]),
    include_dirs=[numpy.get_include()],
) 

and the file a.py would contain the following lines in the header:

try:
    import cython_my_module.b
except ImportError:
    import b

The way it works is very simple: if you don't do anything (ie if you don't compile the cython files) then module a.py imports module b.py ; however, if you run python setup.py build_ext --inplace then the compiled cython files will appear inside cython_my_module and the next time you run a.py it will automatically import the cython module b.pyx (actually it will import the compiled library b.so ).

So far it seems to work and requires almost no effort. Hope it helps.

fish2000 solution seems more generic but I haven't tried it yet.