Cythonize python modules with same name under different packages

Question

I am cythonizing some of my python modules within a project.

python 3.7.*
Ubuntu (different versions depending on docker images and dev environments)
Cython==0.29.14

I'm running into an issue where if I have modules with the same name in different packages all of them get transpiled to .c but only one of them gets compiled with resulting .so file. for one of them the .so file is generated in the current directory instead of --inplace alongside the source file.

Folder structure:

    .  
    ├── my_pkg  
    │   ├── training  
    │   │   └── util.py  
    │   └── validation  
    │       └── util.py  
    └── compile.py  

compile.py looks like this:

from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension

ext_modules = [
    Extension("*",
        ["./my_pgk/*/*.py"]
    )
]

setup(
    name="my_project",
    ext_modules=cythonize(
        ext_modules,
        compiler_directives={'language_level' : "3"}),
)

Execution:

python compile.py build_ext --inplace

EDIT : Looking at (part of) the command log I notice that for one module the .so file is actually generated but not "--inplace" alongside the source python&c files but rather in the current directory. Tried it with more than two modules and indeed for all but the last one the .so is placed correctly but for the last one it is in the current directory.

Output:

building 'my_pgk.training.utils' extension
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/./src/app/training/utils.o -o ./my_pgk/training/utils.cpython-37m-x86_64-linux-gnu.so

building 'utils' extension
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/./my_pgk/validation/utils.o -o ./utils.cpython-37m-x86_64-linux-gnu.so

Folder structure after run:

    .  
    ├── my_pkg  
    │   ├── training  
    │   │   ├── util.py
    │   │   ├── util.c
    │   │   └── utils.cpython-37m-x86_64-linux-gnu.so
    │   └── validation  
    │       ├── util.py
    │       └── util.c  
    ├── compile.py  
EDIT└── utils.cpython-37m-x86_64-linux-gnu.so // this is misplaced

This would generate a .c and an .so file alongside the original .py file except when there are modules with the same name even though they are in different packages.

I want to have the .so files for all .py files, obviously

I've tried listing them explicitly by name in the ext_modules list instead of a glob but that didn't do it either.

For now what I end up doing is run cythonize more than once with different ext_module lists that don't include duplicate module names , specifying the modules individually with intended target locations, but that's not sustainable.

How should I specify my modules and/or what options should I include to get all the .so files generated in one run? I imagine I am doling something wrong because there's no requirement for global module-name uniqueness among all (sub)packages and so it should be a usecase that is supported.

Answer 1

Looking at the code for cythonize at the cython repo on github I realized what was happening.

Basically, if the --inplace flag is used and if the target location/name of the module is not recognized as a package, ie a folder that contains an __init__.py[c|x|d] file, then cythonize walks up the supplied target path string until it finds a package or until the path is "exhausted" in which case it just saves the generated file in the current directory.

So what happened was indeed that in one particular package I was missing a __init__.py file and that's why the .so was "misplaced". I just failed to recognize that it was always the same module that was failing.

The simplified example that I provided above was not reflecting the actual situation since no init files were present