Trouble organizing Python library for import to behave in expected manner

Question

I've had a lot of trouble figuring out a key point about how the import mechanism works, and how this relates to organizing packages.

Suppose I've written two or more unrelated, reusable libraries. (I'll use "library" informally as a collection of code and resources, including tests and possibly data, as opposed to a "package" in the formal Python sense.) Here are two imaginary libraries in a parent directory called "my_libraries":

my_libraries/
├── audio_studio
│   ├── src
│   │   ├── distortion.py
│   │   ├── filter.py
│   │   └── reverb.py
│   └── test
│       └── test_audio.py
└── picasso_graphics
    ├── src
    │   ├── brushes.py
    │   ├── colors.py
    │   └── easel.py
    └── test
        └── test_picasso.py

I'm hoping to accomplish all three of the following, all of which seem to me to be normal practice or expectation:

1. MAIN LIBRARY CODE IN SUBDIRECTORY

For neatness of library organization, I want to put the library's core code in a subdirectory such as "src" rather than at the top-level directory. (My point here isn't to debate whether "src" in particular is a good naming approach; I've read multiple pages pro and con. Some people appear to prefer the form foo/foo, but I think I'd have the same problem I'm describing with that too.)

2. ADD TO $PYTHONPATH JUST ONCE

I'd like to be able to add "my_libraries" to $PYTHONPATH or sys.path just once. If I add a new library to "my_libraries", it's automatically discoverable by my scripts.

3. NORMAL-LOOKING import STATEMENTS

I'd like to be able import from these libraries into other projects in a normal-looking way, without mentioning the "src" directory:

   import picasso_graphics.brushes
   OR
   from picasso_graphics import brushes

HOW TO DO THIS?

Despite much reading and experimentation, I haven't been able to find a solution which satisfies all three of these criteria. The closes I've gotten is to create a picasso_graphics/__init__.py file containing the following:

   base_dir = os.path.dirname(__file__)
   src_dir = os.path.join(base_dir, "src")
   sys.path.insert(0, src_dir)

This almost does what I want, but I have to break up the imports into two statements, so that the __init__.py file executes with the first import:

   import picasso_graphics
   import brushes

Am I making a wrong assumption here about what's possible? Is there a solution which satisfies all three of these criteria?

Answer 1

What you want, Sean, is most likely what is called a namespace project ; Use a tool called pyscaffold to help with writing the boilerplate. Each project should have a setup.cfg with all your project dependencies. Once you have that, create a virtual environment for your project, then install each with... (inside the environment).

pip install -e audio-studio

pip install -e picasso-graphics

Installing your project into your virtual environment will cause your imports to behave as you want -- between projects.

This is a bit of overhead to get started, I know, but these are skills you want to have sooner or later. Setup.cfg, virtual environments, and pip install -e is a magical pattern that just makes things work where other approaches will drive you mad.

Below is a simple example project I created using pyscaffold. Notice there is a package below src and that src does not have an init .py. That is a decision made by the pyscaffold folks to help ease import confusion - you should likely adopt it.

    my_libraries/
├── audio-studio
│   ├── AUTHORS.rst
│   ├── CHANGELOG.rst
│   ├── LICENSE.txt
│   ├── README.rst
│   ├── requirements.txt
│   ├── setup.cfg
│   ├── setup.py
│   ├── src
│   │   └── audio_studio
│   │       ├── __init__.py
│   │       └── skeleton.py
│   └── tests
│       ├── conftest.py
│       └── test_skeleton.py
└── picasso-graphics
    ├── AUTHORS.rst
    ├── CHANGELOG.rst
    ├── LICENSE.txt
    ├── README.rst
    ├── requirements.txt
    ├── setup.cfg
    ├── setup.py
    ├── src
    │   └── picasso_graphics
    │       ├── __init__.py
    │       └── skeleton.py
    └── tests
        ├── conftest.py
        └── test_skeleton.py