导入在解释器启动时不可用的Python模块

Question

I have code which looks like this:

import sys, importlib

try:
    import pip
except ImportError:
    raise ImportError("Please install pip")

reqs = ["sh", "vcstools"]
for req in reqs:
    sys.stdout.write("checking for %s..." % req)
    try:
        importlib.import_module(req)
        print("found")
    except ImportError:
        print("missing!")
        print("Installing %s..." % req)
        pip.main(['install', '--user', req])
        #importlib.invalidate_caches() python3 only
        # XXX fails
        importlib.import_module(req)
        new_mods = True

    locs = locals()
    locs[req] = sys.modules[req]

print(sh, vcstools)

This is designed to download (rough) dependencies at runtime and import them (yes, I know, it does not respect version numbers, and I could have used virtualenv, etc. These are separate issues).

If we run this (with nothing installed in ~/.local), we get the following:

checking for sh...missing!
Installing sh...
Downloading/unpacking sh
....
Successfully installed sh
Cleaning up...
Traceback (most recent call last):
  File "test.py", line 20, in <module>
    importlib.import_module(req)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named sh

So we see that after installing the sh module, we cannot immediately import it. If we re-run the script, it will succeed, finding the sh installed in the previous run.

What I find odd is that, we do not see the same behavior for vcstools ; it installs and imports just fine in the same run. What gives? Is there something special about sh ?

Here is the full output from the second run. Notice that we pick up sh from the last run, we then install vcstools and import it without error:

checking for sh...found
checking for vcstools...missing!
Installing vcstools...
Downloading/unpacking vcstools
  Downloading vcstools-0.1.33.tar.gz
  Running setup.py egg_info for package vcstools

Downloading/unpacking pyyaml (from vcstools)
  Downloading PyYAML-3.11.tar.gz (248Kb): 248Kb downloaded
  Running setup.py egg_info for package pyyaml

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python2.7/site-packages (from vcstools)
Installing collected packages: vcstools, pyyaml
  Running setup.py install for vcstools

  Running setup.py install for pyyaml
    checking if libyaml is compilable
    cc -pthread -fno-strict-aliasing -O2 -pipe -DNDEBUG -O2 -pipe -fPIC -fPIC -I/usr/local/include/python2.7 -c build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c -o build/temp.openbsd-5.5-amd64-2.7/check_libyaml.o
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c: In function 'main':
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: for each function it appears in.)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)

    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Successfully installed vcstools pyyaml
Cleaning up...
(<module 'sh' (built-in)>, <module 'vcstools' from '/home/edd/.local/lib/python2.7/site-packages/vcstools/__init__.pyc'>)

This is Python-2.7 on OpenBSD.

Cheers

EDIT: Just noticed the new_mods line is redundant. I will leave it there so the line numbers in the output are not skewed.

Answer 1

This probably is the first package you try to install into ~/.local/... - that means that ~/.local/... is only created with pip.main(['install', '--user', req])

If ~/.local does not exist upon python startup, it is not added to sys.path, and the module will not be found (not even with importlib.invalidate_caches() ).

So you have to add that path into sys.path, or reload sys.path as indicated in this answer , then it will work...

My solution:

import pip
import importlib
import sys

def pip_import(module_name, pip_package=None):
    try:
        mod = importlib.import_module(module_name)
    except ImportError:
        pip.main(['install', '--user', pip_package if pip_package else module_name])

        # If the install path did not exist on start up it has 
        # to be added to sys.path
        if not pip.locations.user_site in sys.path:
            sys.path.append(pip.locations.user_site)

        mod = importlib.import_module(module_name)

    # Add the imported module to the global namespace so it
    # can be used just like `import module`
    globals()[module_name] = mod