Installing python packages locally doesn't always work

Question

I'm creating a python 3.9 program and want to install packages locally. So the way my project is set up is this:

__main__.py
test.py
requirements.txt
lib/
    __init__.py

In my requirements.txt file I have 3 lines:

colorama==0.2.2
click==8.0.3
pendulum==2.1.2

Then I run: python -m pip install -r requirements.txt -t ./lib This installs all the packages and dependencies inside of the lib directory.

Then I import the modules at the top of my test.py file:

from lib import colorama
from lib import click
from lib import pendulum

In doing some testing, I've found that colorama works fine. I'll use it in a simple test: print(colorama.Fore.BLUE + "Hello, World!") . The text is blue in the console and everything is working.

I then try to use the other packages and I get ModuleNotFoundError exception: print(pendulum.now('Europe/Paris'))

Exception has occurred: ModuleNotFoundError - No module named 'pendulum' This is coming from one of its own files.

The same thing happens when I use Click, but it's a little different. I'll get the same ModuleNotFound exception, but it's for its own dependency on Colorama. I don't think it's related to the fact that I'm also importing Colorama because if I uninstall I get the same error.

I've also tried this with the python-docx package. I added python-docx==0.8.11 to the requirements.txt file, then issued the same command as above to install to my local lib directory. It seems to install fine. I see the docx directory and all its dependencies. Then I import from lib import docx then do something simple in test.py: doc = docx.Document()

Then get ModuleNotFound error: File "C:\\Users\\name\\Development\\python\\test-local-package\\lib\\docx_ init _.py", line 3, in (Current frame) No Module named 'docx'

Does anyone know what I'm doing wrong?

Answer 1

When you put those libraries into your lib folder and import them the way you are doing, you're changing their package names. No longer is colorama a top-level package, it's now lib.colorama . Some libraries might be fine with that, but for others, they expect to be able to import their own code using their normal names. If colorama.some_submodule tries to import colorama , it will fail.

It's important to realize that a statement like from lib import colorama doesn't change how colorama can be found everywhere. It only changes the local namespace. The package is still lib.colorama , we've just bound it to the name colorama in the current module.

As JonSG has suggested in comments, a better solution is to put the lib folder into the Python search path so that import colorama will find the package with its normal name. Modifying sys.path is one way to do that, another is the PYTHONPATH environment variable (probably not ideal for your current issue, but sometimes useful in other situations).