Build and Deliver Python modules using Cythonize in Setuptools

Question

I am building a Python package after compilation using Cython and cythonize.

Here is an example of how my setup.py looks like:

from setuptools import setup, find_packages, Extension
from Cython.Build import cythonize


setup(
    name="My_Pkg",
    version="0.0",
    ext_modules=cythonize('lib/my_pkg/my_mod.py'),
)

And here is and example of how my project directory tree looks like:

.
├── lib
│   └── my_pkg
│       ├── __init__.py
│       └── my_mod.py
└── setup.py

When executing setuptools, everithing works fine:

MACOSX_DEPLOYMENT_TARGET=10.15.1 python setup.py build_ext bdist_wheel

My modules are correctly converted to '.c' and compiled to '.so' in the 'build' directory. When everything is completed, wheel is created in 'dist':

.
├── My_Pkg.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── build
│   ├── bdist.macosx-10.7-x86_64
│   ├── lib.macosx-10.7-x86_64-3.7
│   │   └── my_pkg
│   │       └── my_mod.cpython-37m-darwin.so
│   └── temp.macosx-10.7-x86_64-3.7
│       └── lib
│           └── my_pkg
│               └── my_mod.o
├── dist
│   └── My_Pkg-0.0-cp37-cp37m-macosx_10_7_x86_64.whl
├── lib
│   └── my_pkg
│       ├── __init__.py
│       ├── my_mod.c
│       └── my_mod.py
└── setup.py

My problem is the I'd like the '__init__.py' to be copied in the 'build' as well (without being compiled) during the setuptools build phase. I also would like it to be packaged in the generated wheel as a part of the distribution.

Any idea?

Answer 1

To get this to work I had to add two keyword arguments to setup in setup.py :

packages=["my_pkg"],
package_dir={'': 'lib'},do two things:

(you could probably also use find_packages . This is how you tell setuptools about the Python packages it should be installing. Specifically we tell it that we have a package called "my_pkg" and it should look in lib for them. The best place to read about this is the distutils documentation (setuptools builds on distutils but mostly documents the changes).

Note that this doesn't include the.py files in the "build" folder. This is purely a temporary working space so it doesn't make sense to pointlessly copy files to it. You shouldn't be trying to import modules stashed in the "build" folder They are included in the wheel, which is the important point for being able to distribute them.

Answer 2

Thanks to the help form comments and this other post I came out with a solution using the following 'setup.py':

from setuptools import setup
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize


class build_py(build_py_orig):

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return [(pkg, mod, file, ) for (pkg, mod, file, ) in modules if mod == '__init__']


setup(
    name="My_Pkg",
    version="0.0",
    ext_modules=cythonize('lib/my_pkg/my_mod.py'),
    packages=["my_pkg"],
    package_dir={'': 'lib'},
    cmdclass={'build_py': build_py},
)