Python accessing modules from package that is distributed over different directories

Question

I have a question regarding one single module that is distributed over multiple directories.

Let's say I have these two file and directories:

~/lib/python
   xxx
      __init__.py
      util
         __init__.py
         module1.py
         module2.py
~/graphics/python
   xxx
      __init__.py
      misc
         __init__.py
         module3.py
         module4.py

So then in my Python modules, I did this:

import sys
pythonlibpath = '~/lib/python'
if pythonlibpath not in sys.path: sys.path.append(pythonlibpath)
import xxx.util.module1

which works.

Now, the problem is that I need xxx.misc.module3, so I did this:

import sys
graphicslibpath = '~/graphics/python'
if graphicslibpath not in sys.path: sys.path.append(graphicslibpath)
import xxx.misc.module3

but I get this error:

ImportError: No module named misc.module3

It seems like it somehow still remembers that there was a xxx package in ~/lib/python and then tries to find misc.module3 from there.

How do I get around this issue?

Answer 1

You can't without an extreme amount of trickery that pulls one package structure into the other. Python requires that all modules in a package be under a single subdirectory. See the os source to learn how it handles os.path .

Answer 2

This is addressed by Implicit Namespace Packages in Python 3.3. See PEP-420 .

Answer 3

Python does indeed remember that there was a xxx package. This is pretty much necessary to achieve acceptable performance, once modules and packages are loaded they are cached. You can see which modules are loaded by looking the the dictionary sys.modules .

sys.modules is a normal dictionary so you can remove a package from it to force it to be reloaded like below:

import sys
print sys.modules
import xml
print sys.modules
del sys.modules['xml']
print sys.modules

Notice that after importing the xml package it is the dictionary, however it is possible to remove it from that dictionary too. This is a point I make for pedagogical purposes only, I would not recommend this approach in a real application . Also if you need to use your misc and util packages together this would not work so great. If at all possible rearrange your source code structure to better fit the normal Python module loading mechanism.

Answer 4

This is an adaptation of an answer to a similar question .

Following up on @Gary's answer, the PEP 420 page says to use the following code on shared __init__.py packages.

__init__.py :

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

This code should be placed inside the xxx directory's __init__.py . See the *s below

someroot/
├── graphics
│   └── python
│       └── xxx
│           ├── ****__init__.py****
│           └── misc
│               ├── __init__.py
│               ├── module3.py
│               └── module4.py
└── lib
    └── python
        └── xxx
            ├── ****__init__.py****
            └── util
                ├── __init__.py
                ├── module1.py
                └── module2.py

Some setup.sh file to add to the Python Path:

libPath=someroot/lib/python/
graphicsPath=someroot/graphics/python/
export PYTHONPATH=$PYTHONPATH:$libPath:$graphicsPath

Python test code (tested on Python versions 2.7.14 and 3.6.4 using pyenv ):

import xxx.util.module1
import xxx.misc.module3 # No errors