Running pip install requirements.txt fails in docker build

Question

I am trying to create a docker image for machine learning model but it fails if i try to install from requirements.txt but if I run below docker file then it runs successfully

The following is my docker file:

FROM alpine:latest

RUN mkdir -p /home/ePortalService
WORKDIR /home/ePortalService

COPY requirements.txt /home/ePortalService

RUN apk --update add --virtual scipy-runtime python py-pip \
    && apk add --virtual scipy-build \
        build-base python-dev openblas-dev freetype-dev pkgconfig gfortran lapack-dev \
    && ln -s /usr/include/locale.h /usr/include/xlocale.h \
    && pip install --no-cache-dir numpy \
    && pip install --no-cache-dir pandas \
    && pip install --no-cache-dir matplotlib>=3.0.0 \
    && pip install --no-cache-dir scipy \
    && apk del scipy-build \
    && apk add --virtual scipy-runtime \
        freetype libgfortran libgcc libpng  libstdc++ musl openblas tcl tk \
    && rm -rf /var/cache/apk/*

If I try to manually run pip install numpy, then all the way to pip install scipy it works.

I eventually intend to make this requirements.txt be part of a docker image's Dockerfile build script (without using virtualenv inside the docker image) but this throws error

Any ideas why it works manually but not automatically, and how to fix it?

the following is the error log:

Sending build context to Docker daemon  421.4MB
Step 1/6 : FROM alpine:latest
 ---> cc0abc535e36
Step 2/6 : RUN mkdir -p /home/ePortalService
 ---> Running in 8f47d4b7dea3
Removing intermediate container 8f47d4b7dea3
 ---> b0bc3aacada2
Step 3/6 : WORKDIR /home/ePortalService
 ---> Running in 84378138f606
Removing intermediate container 84378138f606
 ---> 3f907a00b6ed
Step 4/6 : COPY requirements.txt /home/ePortalService
 ---> 8c691dc5aaaf
Step 5/6 : RUN apk --update add --virtual scipy-runtime python py-pip     && apk add --virtual scipy-build         build-base python-dev openblas-dev freetype-dev pkgconfig gfortran lapack-dev     && ln -s /usr/include/locale.h /usr/include/xlocale.h     && pip install --no-cache-dir -r requirements.txt     && apk del scipy-build     && apk add --virtual scipy-runtime         freetype libgfortran libgcc libpng  libstdc++ musl openblas tcl tk     && rm -rf /var/cache/apk/*
 ---> Running in 185848aecaaa
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/13) Installing libbz2 (1.0.8-r1)
(2/13) Installing expat (2.2.9-r1)
(3/13) Installing libffi (3.2.1-r6)
(4/13) Installing gdbm (1.13-r1)
(5/13) Installing ncurses-terminfo-base (6.1_p20191130-r0)
(6/13) Installing ncurses-terminfo (6.1_p20191130-r0)
(7/13) Installing ncurses-libs (6.1_p20191130-r0)
(8/13) Installing readline (8.0.1-r0)
(9/13) Installing sqlite-libs (3.30.1-r1)
(10/13) Installing python2 (2.7.16-r3)
(11/13) Installing py-setuptools (42.0.2-r0)
(12/13) Installing py2-pip (18.1-r0)
(13/13) Installing scipy-runtime (20200107.074517)
Executing busybox-1.31.1-r8.trigger
OK: 65 MiB in 27 packages
(1/35) Installing libgcc (9.2.0-r3)
(2/35) Installing libstdc++ (9.2.0-r3)
(3/35) Installing binutils (2.33.1-r0)
(4/35) Installing libmagic (5.37-r1)
(5/35) Installing file (5.37-r1)
(6/35) Installing gmp (6.1.2-r1)
(7/35) Installing isl (0.18-r0)
(8/35) Installing libgomp (9.2.0-r3)
(9/35) Installing libatomic (9.2.0-r3)
(10/35) Installing mpfr4 (4.0.2-r1)
(11/35) Installing mpc1 (1.1.0-r1)
(12/35) Installing gcc (9.2.0-r3)
(13/35) Installing musl-dev (1.1.24-r0)
(14/35) Installing libc-dev (0.7.2-r0)
(15/35) Installing g++ (9.2.0-r3)
(16/35) Installing make (4.2.1-r2)
(17/35) Installing fortify-headers (1.1-r0)
(18/35) Installing build-base (0.5-r1)
(19/35) Installing pkgconf (1.6.3-r0)
(20/35) Installing python2-dev (2.7.16-r3)
(21/35) Installing libquadmath (9.2.0-r3)
(22/35) Installing libgfortran (9.2.0-r3)
(23/35) Installing gfortran (9.2.0-r3)
(24/35) Installing openblas-ilp64 (0.3.7-r2)
(25/35) Installing openblas (0.3.7-r2)
(26/35) Installing openblas-dev (0.3.7-r2)
(27/35) Installing libpng (1.6.37-r1)
(28/35) Installing freetype (2.10.1-r0)
(29/35) Installing zlib-dev (1.2.11-r3)
(30/35) Installing libpng-dev (1.6.37-r1)
(31/35) Installing freetype-dev (2.10.1-r0)
(32/35) Installing blas (3.9.0-r1)
(33/35) Installing lapack (3.9.0-r1)
(34/35) Installing lapack-dev (3.9.0-r1)
(35/35) Installing scipy-build (20200107.074518)
Executing busybox-1.31.1-r8.trigger
OK: 326 MiB in 62 packages
Collecting Click (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting cycler (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting Flask (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
Collecting itsdangerous (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Jinja2 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl (125kB)
Collecting joblib (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/28/5c/cf6a2b65a321c4a209efcdf64c2689efae2cb62661f8f6f4bb28547cf1bf/joblib-0.14.1-py2.py3-none-any.whl (294kB)
Collecting kiwisolver (from -r requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/16/e7/df58eb8868d183223692d2a62529a594f6414964a3ae93548467b146a24d/kiwisolver-1.1.0.tar.gz
Collecting MarkupSafe (from -r requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/b9/2e/64db92e53b86efccfaea71321f597fa2e1b2bd3853d8ce658568f7a13094/MarkupSafe-1.1.1.tar.gz
Collecting mlxtend (from -r requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/52/04/c362f34f666f0ddc7cf593805e64d64fa670ed96fd9302e68549dd48287d/mlxtend-0.17.0-py2.py3-none-any.whl (1.3MB)
Collecting pandas (from -r requirements.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz (11.8MB)
    Complete output from command python setup.py egg_info:
    WARNING: The wheel package is not available.
      Failed building wheel for numpy
      Failed cleaning build dir for numpy
    ERROR: Failed to build one or more wheels
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-9fKYgP/pandas/setup.py", line 746, in <module>
        **setuptools_kwargs)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/usr/bin/python2', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp6GfJkV', '--quiet', 'numpy>=1.12.0']' returned non-zero exit status 1

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-9fKYgP/pandas/
The command '/bin/sh -c apk --update add --virtual scipy-runtime python py-pip     && apk add --virtual scipy-build         build-base python-dev openblas-dev freetype-dev pkgconfig gfortran lapack-dev     && ln -s /usr/include/locale.h /usr/include/xlocale.h     && pip install --no-cache-dir -r requirements.txt     && apk del scipy-build     && apk add --virtual scipy-runtime         freetype libgfortran libgcc libpng  libstdc++ musl openblas tcl tk     && rm -rf /var/cache/apk/*' returned a non-zero code: 1

I am not able to find the actual reason as what mistake is been made here

Answer 1

Try the below and install wheel

FROM alpine:latest

RUN mkdir -p /home/ePortalService
WORKDIR /home/ePortalService

COPY requirements.txt /home/ePortalService

RUN apk --update add --virtual scipy-runtime python py-pip \
    && apk add --virtual scipy-build \
        build-base python-dev openblas-dev freetype-dev pkgconfig gfortran lapack-dev \
    && ln -s /usr/include/locale.h /usr/include/xlocale.h \
    && pip install --no-cache-dir -U pip wheel setuptools \
    && pip install --no-cache-dir requirements.txt \
    && apk del scipy-build \
    && apk add --virtual scipy-runtime \
        freetype libgfortran libgcc libpng  libstdc++ musl openblas tcl tk \
    && rm -rf /var/cache/apk/*```

Answer 2

I think that the correct answer is that you need to be aware that the dockerfile run commands are in the context of the container image.

So, when you run pip install -r requirements.txt the file path of requirements.txt is invisible to pip.

What you need to do is to copy requirements.txt into the container, using:

COPY filepathonhost filepathinsideimage,

and then supply the file path inside the image to pip.

This is documented at the following link... https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#add-or-copy

Answer 3

The version of python-setuptools in the package repository might be out-of-date / unmaintained. Upgrade it with pip instead of relying on the package manager, add this step in you dockerfile in which you pip install --upgrade setuptools before installing python packages.

Answer 4

You could use slim images :

Consider this Dockerfile:

FROM python:3.7-slim
RUN pip install numpy pandas scipy matplotlib

with resulting size: 504MB

Though this is not the smallest possible size, in this case, you wouldn't need to find dependencies (sometimes it can be really hard), especially for math libraries.