Docker 容器中缺少 python 包

Question

I am working on a tensorflow notebook in a Docker container. I manage additional dependencies via pipenv which I install during the build phase of the container. Strangely enough some packages, which I explicitly install, "certifi" for example, do not seem to show up in the python environment.

I already tried to install it with pip in a RUN command, switched Python versions of the Pipfile and added pretty much everything to the PATH and PYTHONPATH env variables. Strangely enough, if I log in to the container and do it by hand it works like a charm.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-06bc95e4104d> in <module>
      6 # data wrangling
      7 import json
----> 8 import spacy
      9 from langdetect import detect
     10 import psycopg2

~usr/local/lib/python3.5/dist-packages/spacy/__init__.py in <module>
     10 from thinc.neural.util import prefer_gpu, require_gpu
     11 
---> 12 from .cli.info import info as cli_info
     13 from .glossary import explain
     14 from .about import __version__

~usr/local/lib/python3.5/dist-packages/spacy/cli/__init__.py in <module>
----> 1 from .download import download  # noqa: F401
      2 from .info import info  # noqa: F401
      3 from .link import link  # noqa: F401
      4 from .package import package  # noqa: F401
      5 from .profile import profile  # noqa: F401

~usr/local/lib/python3.5/dist-packages/spacy/cli/download.py in <module>
      3 
      4 import plac
----> 5 import requests
      6 import os
      7 import subprocess

~usr/local/lib/python3.5/dist-packages/requests/__init__.py in <module>
    110 from .__version__ import __copyright__, __cake__
    111 
--> 112 from . import utils
    113 from . import packages
    114 from .models import Request, Response, PreparedRequest

~usr/local/lib/python3.5/dist-packages/requests/utils.py in <module>
     22 
     23 from .__version__ import __version__
---> 24 from . import certs
     25 # to_native_string is unused here, but imported here for backwards compatibility
     26 from ._internal_utils import to_native_string

~usr/local/lib/python3.5/dist-packages/requests/certs.py in <module>
     13 packaged CA bundle.
     14 """
---> 15 from certifi import where
     16 
     17 if __name__ == '__main__':

ImportError: No module named 'certifi'

Did somebody face a similar problem already?

The minimal setup for reproduction would be the following:

docker-compose.yaml:

version: '3.1'
services:

  notebook:
    container_name: Notebook
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "5430:8888"
    user: ${CURRENT_UID}

Dockerfile:

FROM tensorflow/tensorflow:latest-py3-jupyter

RUN pip install --user pipenv
ENV PATH="/root/.local/bin:/.local/bin:${PATH}"
ENV PYTHONPATH="/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/.local/lib/python3.5/site-packages:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages:${PYTHONPATH}"

RUN apt-get update
RUN apt-get install -y libsm6 libxext6 libxrender-dev libglib2.0-0 postgresql libpq-dev

# We copy just the requirements.txt first to leverage Docker cache
COPY ./Pipfile ./Pipfile
COPY ./Pipfile.lock ./Pipfile.lock

RUN pipenv install --deploy --system --ignore-pipfile
RUN python -m spacy download xx

RUN pip install --user certifi

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
scikit-learn = {extras = ["alldeps"]}
spacy = "*"
langdetect = "*"
ipykernel = "*"
bokeh = "*"
ipywidgets = "*"
plotly = "*"
lapjv = "*"
tensorflow = "*"
"psycopg2" = "*"
gensim = "*"
cmake = "*"
certifi = "*"

[dev-packages]

[requires]
python_version = "3.5"

Locking the Pipfile with pipenv lock , building and bringing the container up with docker-compose up --build and logging into the container reveals that the certifi package is not installed. Help would be greatly appreciated!

Answer 1

From your dockerfile, it looks like certifi is being installed after you run spacy which from the stack trace seems to rely on certifi

RUN python -m spacy download xx

RUN pip install --user certifi

switch the order of those lines to install it before running the script which relies on it.

Also, make sure that everything is configured to use python3, or explicitly use python3 and pip3

Answer 2

Reviving the dead question: Apparently there was a version conflict that pip was not able to resolve. I switched to using Poetry as a module management solution and circumvented the issua that way.