Python namespace package clear folder structure

Question

I would like to set up a python namespace package containing several connected packages which need to be installable independently unless depencies are explicitly specified. Existing solutions however seem more ore less messy to me.

One of the packages contains moste of the problems logic for example and the others contain auxiliray functionality such as plotting and data export. The logic package needs to stay slim and can not import more than numpy where as the other packages can utilise more complex packages like pandas and matplolib . I would like to set up a package structure which looks like the resulting namespace of a namespace package but without unnecessary folder nesting something like this:

namespace
├── logic
│    ├── __init__.py
|    ├──  functions.py
│    └──  setup.py # requires numpy    
├── datastructure
│    ├── __init__.py
|    ├──  functions.py
│    └──  setup.py # requires namespace.logic and pandas
├── plotting
│    ├── __init__.py
|    ├──  functions.py
│    └──  setup.py # requires namespace.logic, namespace.datastructure and matplotlib
└── setup.py #should install every package in namespace

I figured that this looks like a conventional package with modules but I did not find a way yet to set it up as a packgae while mainintainign the option to only install specific modules therefore I assumed a namespace package should offer that option but I can not quite get it to work with pip

At the moment I would be required to have two more directory levels like this:

namespace
├── NamespaceLogic          #don't want this
│   ├── namespace           #don't want this
│   │   └── logic
│   │       └── __init__.py
│   └── setup.py
├── NamespaceDatastructure  #don't want this
│   ├── namespace           #don't want this
│   │   └── datastructure
│   │       └── __init__.py
│   └── setup.py
├── NamespacePlotting       #don't want this
│   ├── namespace           #don't want this
│   │   └── plotting
│   │       └── __init__.py
│   └── setup.py
└── setup.py

My problem is similar to this question: Python pip install sub-package from own package but I would like to avoid having to many subfolders as this poses the risk to max out the path length restrictions of my system (+ it confuses everyone else). How do I need to configure the different setup.py files in order to be able to run

pip install namespace #installs namespace.logic, namespace.datastructure, namespace.plotting
pip install namespce.logic #installs only namspace.logic and works in an environment with numpy which does not have pandas or matplotlib

Answer 1

You can use the package_dir option of setuptools to your advantage to get rid of the empty folders for the namespace packages:

NmspcPing
├── ping
│   └── __init__.py
└── setup.py

import setuptools
setuptools.setup(
    name='NmspcPing',
    version='0.0.0.dev0',
    packages=['nmspc.ping'],
    package_dir={'nmspc.ping': 'ping'},
)

---

Something like the following would also be feasible, but depending on how the projects are built or installed, the setup.py files might be included as part of the packages as well (which is probably unwanted):

.
├── ping
│   ├── __init__.py
│   └── setup.py
├── pong
│   ├── __init__.py
│   └── setup.py
└── setup.py

---

If the path length restriction is an issue, then using shorter package names might be a better bet. Because in many cases the packages are installed with all the directory levels anyway (unless they stay zipped), even if you skip them in your source code repository.

Honestly I would be surprised if a path length restriction issue actually happened, and I believe it would anyway still happen on things that you don't have control over (like 3rd party packages: numpy, pandas, plotlib probably have lots of nested sub-packages as well).