Tensorflow Python 3 Docker Image with mysql failing to install

Question

Note: yes, there are many, many posts regarding the issues of installing mysql for python, specifically for python3 (eg Troubles installing mysqlclient with pip3 ). Most solutions are done in a non-dockerized manner. I have also read the mysqlclient install guide. I am still struggling to get it to work

A repo to clone and test for convenience.

Here I am trying to install it to my docker image.

Dockerfile.ai

FROM tensorflow/tensorflow:1.15.0-gpu-py3-jupyter


# --- Install any needed packages specified in requirements.apt
# from SO answer
# RUN apt-get update && apt-get install -y mysql-client && rm -rf /var/lib/apt

COPY requirements.apt .
RUN apt-get update && xargs apt-get install -y < requirements.apt

# --- Install any needed packages specified in requirements.pip
COPY requirements.pip .
RUN pip install --trusted-host pypi.python.org -r requirements.pip

# activate jupyter extensions
RUN jupyter contrib nbextension install \
  && jupyter nbextension enable codefolding/main \
  && jupyter nbextension enable collapsible_headings/main

docker-compose.yml

version: '3'

services:
  tf:
    volumes:
      - ./notebooks:/tf/notebooks
      - ./data:/tf/data
      - ./experiments:/tf/experiments
      - ./models:/tf/models


    ports:
      - '8888:8888'

    build:
      context: .
      dockerfile: Dockerfile.ai


volumes:
  notebooks:
  data:
  experiments:
  models:

requirements.apt

git
python-pip
mysql-server 
python3-dev # from https://pypi.org/project/mysqlclient/
build-essential # from SO solution
default-libmysqlclient-dev # from another SO solution
libmysqlclient-dev

requirements.pip

# This file is used for multiple docker images which may have different bases.

# data related
pymongo
sqlalchemy
# MySQLdb
mysqlclient

# deep learning related
tensorflow-gpu==1.15.0
seaborn
numpy
sacred
requests
pandas
scipy
sklearn

# jupyter
ipykernel
jupyter_contrib_nbextensions

Errors

Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from zipp>=0.5->importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.4->nbconvert>=4.2->jupyter_contrib_nbexten
sions->-r requirements.pip (line 26)) (7.2.0)
Building wheels for collected packages: sqlalchemy, mysqlclient, sacred, sklearn, docopt, py-cpuinfo, jupyter-nbextensions-configurator, pyyaml, jupyter-latex-envs
  Building wheel for sqlalchemy (setup.py): started
  Building wheel for sqlalchemy (setup.py): finished with status 'done'
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.12-cp36-cp36m-linux_x86_64.whl size=1216988 sha256=a18ffb30eefe7a430b978e9ae54506ed6fff1197cd8f06f18fc89ab34ffebde1
  Stored in directory: /root/.cache/pip/wheels/ee/33/44/0788a6e806866ae2e246d5cd841d07498a46bcb3f3c42ea5a4
  Building wheel for mysqlclient (setup.py): started
  Building wheel for mysqlclient (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-fhcie_47 --python-tag cp36
       cwd: /tmp/pip-install-o51hii7f/mysqlclient/
  Complete output (31 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/_exceptions.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/compat.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/connections.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/converters.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/cursors.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/release.py -> build/lib.linux-x86_64-3.6/MySQLdb
  copying MySQLdb/times.py -> build/lib.linux-x86_64-3.6/MySQLdb
  creating build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
  running build_ext
  building 'MySQLdb._mysql' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/MySQLdb
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,4,6,'final',0) -D__version__=1.4.6 -I/usr/include/mysql -I/usr/include/python3.6m -c MySQLdb/_mysql.c -o build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o
  x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o -lmysqlclient -lpthread -lz -lm -lrt -latomic -lssl -lcrypto -ldl -o build/lib.linux-x86_64-3.6/MySQLdb/_mysql.cpython-36m-x86_64-linux-gnu.so
  /usr/bin/ld: cannot find -lssl
  /usr/bin/ld: cannot find -lcrypto
  collect2: error: ld returned 1 exit status
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for mysqlclient
  Running setup.py clean for mysqlclient
  Building wheel for sacred (setup.py): started
  Building wheel for sacred (setup.py): finished with status 'done'
  Created wheel for sacred: filename=sacred-0.8.1-py2.py3-none-any.whl size=109604 sha256=9ae1f0e699ca8ffd5bc83691ac8e40682b045d2277da4bddc9d1c639a031c917
  Stored in directory: /root/.cache/pip/wheels/11/a8/f6/1d5f073245cb0a221962713adf81e56c1c9608083f85ecac9b
  Building wheel for sklearn (setup.py): started
  Building wheel for sklearn (setup.py): finished with status 'done'
  Created wheel for sklearn: filename=sklearn-0.0-py2.py3-none-any.whl size=2397 sha256=9b45378895d96c16201b032e0c190415cb91c02dca496e404b4fd3b337c49730
  Stored in directory: /root/.cache/pip/wheels/76/03/bb/589d421d27431bcd2c6da284d5f2286c8e3b2ea3cf1594c074
  Building wheel for docopt (setup.py): started
  Building wheel for docopt (setup.py): finished with status 'done'
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=19852 sha256=a8da8458c371b8bb8ec40f3e18a974e505593b9f913f1960578a8e8e90f76101
  Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
  Building wheel for py-cpuinfo (setup.py): started
  Building wheel for py-cpuinfo (setup.py): finished with status 'done'
  Created wheel for py-cpuinfo: filename=py_cpuinfo-5.0.0-cp36-none-any.whl size=18912 sha256=cf72494546ffaf64bb66b6c08785c37627a31145d48e03b1a270da64b72c9a55
  Stored in directory: /root/.cache/pip/wheels/01/7e/a9/b982d0fea22b7e4ae5619de949570cde5ad55420cec16e86a5
  Building wheel for jupyter-nbextensions-configurator (setup.py): started
  Building wheel for jupyter-nbextensions-configurator (setup.py): finished with status 'done'
  Created wheel for jupyter-nbextensions-configurator: filename=jupyter_nbextensions_configurator-0.4.1-py2.py3-none-any.whl size=466146 sha256=c5ac621237f2bc3174b0ad762dbea00be7688ab71fc72c2eb2d6013c6a759963
  Stored in directory: /root/.cache/pip/wheels/15/df/fe/2a74fe34709e7fdc5ae153a768675d9fda93cc7d5133ed1fb0
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=45519 sha256=f73f83291978cafd58f9bb245844cd4c4fa44da66839312ebcf777e35e865880
  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191
  Building wheel for jupyter-latex-envs (setup.py): started
  Building wheel for jupyter-latex-envs (setup.py): finished with status 'done'
  Created wheel for jupyter-latex-envs: filename=jupyter_latex_envs-1.4.6-py2.py3-none-any.whl size=966916 sha256=84f81480b78b2d1d6db05ce6e6dce61da11482360c3658124c08f9c5f942f791
  Stored in directory: /root/.cache/pip/wheels/0d/71/2a/164491997299b9f2479a251e254323fe35d946779e18f27956
Successfully built sqlalchemy sacred sklearn docopt py-cpuinfo jupyter-nbextensions-configurator pyyaml jupyter-latex-envs
Failed to build mysqlclient
Installing collected packages: mag, sil, parpar, pymongo, sqlalchemy, mysqlclient, pytz, pandas, scipy, seaborn, docopt, jsonpickle, munch, py-cpuinfo, colorama, packaging, smmap2, gitdb2, GitPython, sacred, urllib3, chardet, certifi, requests, joblib, scikit-learn, sklearn, jupyter-contrib-core, pyyaml, jupyter-nbextensions-configurator, jupyter-latex-envs, lxml, jupyter-highlight-selected-word, jupyter-contrib-nbextensions
    Running setup.py install for mysqlclient: started
Running setup.py install for mysqlclient: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-x870urz9/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-o51hii7f/mysqlclient/
    Complete output (31 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/_exceptions.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/compat.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/connections.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/converters.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/cursors.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/release.py -> build/lib.linux-x86_64-3.6/MySQLdb
    copying MySQLdb/times.py -> build/lib.linux-x86_64-3.6/MySQLdb
    creating build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
    running build_ext
    building 'MySQLdb._mysql' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/MySQLdb
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,4,6,'final',0) -D__version__=1.4.6 -I/usr/include/mysql -I/usr/include/python3.6m -c MySQLdb/_mysql.c -o build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o -lmysqlclient -lpthread -lz -lm -lrt -latomic -lssl -lcrypto -ldl -o build/lib.linux-x86_64-3.6/MySQLdb/_mysql.cpython-36m-x86_64-linux-gnu.so
    /usr/bin/ld: cannot find -lssl
    /usr/bin/ld: cannot find -lcrypto
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o51hii7f/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-x870urz9/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
ERROR: Service 'tf' failed to build: The command '/bin/bash -c pip install --trusted-host pypi.python.org -r requirements.pip' returned a non-zero code: 1

updates

1. Found this: https://github.com/postlight/headless-wp-starter/issues/115 and tried adding

# Avoid ERROR: invoke-rc.d: policy-rc.d denied execution of start.
RUN sed -i "s/^exit 101$/exit 0/" /usr/sbin/policy-rc.d

This changed the errors, but fails nonetheless

Answer 1

From "cannot find -lssl; cannot find -lcrypto" when installing mysql-python? it seems you need to install the libssl-dev package through apt