在python 3中使用pip导入运行时安装的模块

Question

I want to install and import Python 3 modules at runtime.

I'm using the following function to install modules at runtime using pip :

def installModules(modules):
    for module in modules:
        print("Installing module {}...".format(module))

        subprocess.call([sys.executable, "-m", "pip", "install", "--user", module])

The module is installed successfully, but I'm not able to import it at runtime, after the installation finishes. So if I do:

modules = [ "wget", "zipfile2" ]
installModules(module)
import wget

I get a ModuleNotFoundError . If, after that, I start another Python 3 session, I am able to use the modules eg wget , which means that the modules have been installed, but they are not available for this current Python 3 session.

Is it possible in Python 3 to install and then import the installed modules in the same Python 3 session ie right after installation?

Thank you!

EDIT:

On a fresh Ubuntu 19.04 install inside VirtualBox, after a sudo apt-get install python3-pip , running the following script:

import os, sys
import subprocess


def installModules(modules):
    for module in modules:
        print("Installing module {}...".format(module))

        subprocess.call([sys.executable, "-m", "pip", "install", "--user", module])

def process():
    modulesToInstall = [ "wget", "zipfile2" ]
    installModules(modulesToInstall)

process()

import wget

def main():
    wget.download("http://192.168.2.234/test/configure.py")

if __name__ == "__main__":
    main()

I get:

user@user-VirtualBox:~$ python3 script.py
Installing module wget...
Collecting wget
Installing collected packages: wget
Successfully installed wget-3.2
Installing module zipfile2...
Collecting zipfile2
  Using cached https://files.pythonhosted.org/packages/60/ad/d6bc08f235b66c11bbb76df41b973ce93544a907cc0e23c726ea374eee79/zipfile2-0.0.12-py2.py3-none-any.whl
Installing collected packages: zipfile2
Successfully installed zipfile2-0.0.12
Traceback (most recent call last):
  File "script.py", line 17, in <module>
    import wget
ModuleNotFoundError: No module named 'wget'

The Python 3 version is:

user@user-VirtualBox:~$ python3 --version
Python 3.7.3

The pip3 version is:

user@user-VirtualBox:~$ pip3 --version
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)

Other info:

user@user-VirtualBox:~$ whereis python3
python3: /usr/bin/python3.7m /usr/bin/python3.7-config /usr/bin/python3.7 /usr/bin/python3 /usr/bin/python3.7m-config /usr/lib/python3.7 /usr/lib/python3.8 /usr/lib/python3 /etc/python3.7 /etc/python3 /usr/local/lib/python3.7 /usr/include/python3.7m /usr/include/python3.7 /usr/share/python3 /usr/share/man/man1/python3.1.gz

Any ideas?

Answer 1

By default, at startup Python adds the user site-packages dir (I'm going to refer to it as USPD ) in the module search paths. But this only happens if the directory exists on the file system (disk) . I didn't find any official documentation to support this statement ¹ , so I spent some time debugging and wondering why things seem to be so weird.

The above behavior has a major impact on this particular scenario ( pip install --user ). Considering the state (at startup) of the Python process that will install modules:

1. USPD exists:

   • Things are straightforward, everything works OK

2. USPD doesn't exist:

   • Module installation will create it
   • But, since it's not in the module search paths, all the modules installed there won't be available for (simple) import statements

When another Python process is started, it will fall under #1.

To fix things, USPD should be manually added to module search paths. Here's how the (beginning of the) script should look like:

import sys
import os
import subprocess
import site

user_site = site.getusersitepackages()
if user_site not in sys.path:
    sys.path.append(user_site)

# ...

@EDIT0 :

¹ I just came across [Python]: PEP 370 -- Per user site-packages directory - Implementation ( emphasis is mine):

The site module gets a new method adduserpackage() which adds the appropriate directory to the search path. The directory is not added if it doesn't exist when Python is started .