add python modules that depend on each other as git submodules

Question

I have these 2 little modules in 2 separate git repositories:

RepoA/A.py

def foo():
    return "Hello World"

RepoB/B.py

import sys
try:
    import A
except ImportError:
    print("Dependency not available!")
    sys.exit(1)

def bar():
    return A.foo() + " EXTENDED!"

You see that B currently assumes A is importable globally (either installed or in the script-execution directory).

Both have an empty __init__py at their root directory to make them importable as submodules.

Now I have another, bigger repository C that needs B. It has both A and B available as git submodules, so the structure virtually looks like this:

RepoC
|---utils/
|   |---__init__.py
|   |---RepoA/
|   |   |---__init__.py
|   |   |---A.py
|   |---RepoB/
|       |---__init__.py
|       |---B.py
|---C.py

RepoC/C.py

import utils.B

print(B.bar())

RepoC/utils/__init__.py

# remove the 1-layer indirection through the submodule
from .RepoB import B

Now this prints Dependency not available! , as expected because A isn't globally available, but rather at a location B could never guess (in this case it would need to do from ..RepoA import A ). If I make A globally available again by adding it to sys.path it will work:

RepoC/utils/__init__.py

import os, sys
import inspect

def fake_install(module_path):
    # the submitted module path is relative to the caller's location
    # therefore get that script's file location first
    caller_module = inspect.getmodule(inspect.stack()[1][0])
    caller_dir = caller_module.__file__.rsplit(os.path.sep, 1)[0]
    # build an absolute file path and append it to the system paths
    path = os.path.join(os.path.abspath(caller_dir), *module_path.split("."))
    sys.path.append(path)

fake_install("RepoA")

# remove the 1-layer indirection through the submodule
from .RepoB import B

This feels like a horrendous solution though.

Another idea was to not use git submodules at all, but rather just collect the dependencies as git-links in a requirements.txt, write some setup.exe scripts and let pip actually install them.

How can I elegantly overcome this problem? Is there any import trickery that lets me do exactly this?

Answer 1

As you may have guessed, I think you have two options: either you let A and B be part of C, or you make B and C standalone packages.

It's the job of pip to "put A somewhere in sys.path", so you might as well let him do this and not do it yourself. You can use git links in requirements but not in setup.py, so if you have more dependencies than that (D requires B which requires C) and you can't publish those on PyPI you'll likely need a private PyPI server (devpi works well for this).

Answer 2

(I just figured it out myself)
This can be solved by removing the extra indirection introduced by the submodule repository's folder. You need the currently empty __init__.py files in RepoA and Repo to act as their containing module (here utils ) to give RepoC the ability to make the dependency available there.

Edit your RepoB/__init__.py to look like this:

from .. import *

Then also make your dependency A available in your utils by adding this to the utils' __init__.py :

from .RepoA import A

Now all you need to do is make the import in B.py local:

from . import A

You can also make it work as a global dependency too:

try:
    from . import A
except ImportError:
    import A

Now all the library's user has to do is either:

• Globally install the dependency, or
• Make the dependency available in the containing module

---

To make this approach more general I would have all the __init__.py in Submodules' root directories just act as bridges like this:

RepoA/__init__.py

from .. import *
__all__ = ["A"]

RepoB/__init__.py

from .. import *
__all__ = ["B"]

and do the same, which is removing the indirection, "from the other side". In this case in utils/__init__.py :

from .RepoA import *
from .RepoB import *