Python Dynamic Import Can't Find Packages in Virtualenv

Question

So, I have a directory structure:

main.py
\_ modules/
   \_ a.py
   \_ b.py

In main.py, modules are dynamically loaded at runtime, depending on which modules are specified. (This allows for a hypothetical c.py to be added, main.py to be rerun, and the program to detect the addition of c.py and run it.)

The problem is that b.py imports a module installed via pip (in a virtualenv). (I'm going to refer to it as a library to help avoid confusion.) When b.py is run directly ( python b.py ) the library imports just fine. When the shell is opened and the library is imported by hand, it works.

But, when main.py is run and b.py is dynamically imported (using pkgutil.iter_modules to detect the modules and then importlib.import_module to import the required ones), the library that b.py imports isn't found - an ImportError: No module is thrown.

To recap: a module imports an installed library, and this works when the module is run directly or when the library in question is imported manually in the python interpreter, but when the module is dynamically imported, the library isn't found. What gives?

Answer 1

A third, unrelated detail was the problem: pyenv.

There's a bug report here , but long story short is that pyenv uses shims to intercept package imports and route them correctly.

This means that pyenv has to muddle with the path. When I run python directly, the shims are apparent in the python path:

[dnelson@blueharvest somedir]$ python
Python 2.7 (r27:82500, Jun  1 2015, 15:01:57) 
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux3
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.getsitepackages()
['/home/dnelson/.pyenv/versions/2.7/lib/python2.7/site-packages', '/home/dnelson/.pyenv/versions/2.7/lib/site-python']
>>> site.PREFIXES
['/home/dnelson/.pyenv/versions/2.7', '/home/dnelson/.pyenv/versions/2.7']
>>> 

However, in the dynamically imported package, the same code results in an output of:

['/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/lib/site-python']
['/usr', '/usr']

So, the issue appears to be that pyenv isn't doing its shim magic during dynamic imports.

To solve the problem, I used pip install --user to force pip to install to the place where the dynamic import is looking, rather than where pyenv wants the install to go. (also add a -I if you've previously installed the package to force a reinstall)

It can also be solved by appending the install location (in my case, /usr/lib/python2.7/site-packages ) to the python path using sys.path.append , but that smells awful and could cause problems down the line for other people.

Answer 2

Virtualenvs are notorious. They tinker around with paths lots of times and mess up quite a lot of things.

You need to check the PATH variable inside b.py in both scenarios. This will most probably not be the same.

You'd need to set the path to include your c.py 's directory. You can check the PATH in main.py and if it is correct there, it would mean that one of your other imports is changing your sys.path ina way to remove that.