Docker 中的 airflow initdb 抛出 ImportError: cannot import name 'import_string'

Question

I am trying to dockerize airflow , my Dockerfile looks like this

FROM python:3.5.2

RUN mkdir -p /src/airflow
RUN mkdir -p /src/airflow/logs
RUN mkdir -p /src/airflow/plugins
WORKDIR /src
COPY . .
RUN pip install psycopg2
RUN pip install -r requirements.txt
COPY airflow.cfg /src/airflow
ENV AIRFLOW_HOME /src/airflow
ENV PYTHONPATH "${PYTHONPATH}:/src"
RUN airflow initdb
EXPOSE 8080
ENTRYPOINT ./airflow-start.sh

while my docker-compose.yml looks like this

version: "3"
services:
  airflow:
    container_name: airflow
    network_mode: host
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 8080:8080

The output of $ docker-compose build comes up like normal, every step executes and then

Step 12/14 : RUN airflow initdb
 ---> Running in 8b7ebe406978
[2020-04-21 10:34:21,419] {__init__.py:45} INFO - Using executor LocalExecutor
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 17, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python3.5/site-packages/airflow/bin/cli.py", line 59, in <module>
    from airflow.www.app import cached_app
  File "/usr/local/lib/python3.5/site-packages/airflow/www/app.py", line 20, in <module>
    from flask_cache import Cache
  File "/usr/local/lib/python3.5/site-packages/flask_cache/__init__.py", line 24, in <module>
    from werkzeug import import_string
ImportError: cannot import name 'import_string'
ERROR: Service 'airflow' failed to build: The command '/bin/sh -c airflow initdb' returned a non-zero code: 1

postgres is running on host system.
I have tried multiple ways but this keeps on happening.
I even tried puckel/docker-airflow image and the same error occurred.
Can someone tell me what am I doing wrong?


Project Structure:

root
  -airflow_dags
  -Dockerfile
  -docker-compose.yml
  -airflow-start.sh
  -airflow.cfg

In case it's relevant: airflow-start.sh

In airflow.cfg :

dags_folder = /src/airflow_dags/
sql_alchemy_conn = postgresql://airflow:airflow@localhost:5432/airflow

Answer 1

If possible get your code running without touching docker... run it directly on your host... of course this means your host ( your laptop or wherever you are executing your commands, could be a remote VPS debian box ) must have the same OS as your Dockerfile, I see in this case FROM python:3.5.2 is actually using debian 8

Short of doing above launch a toy container which does nothing yet executes and lets you login to it to manually run your commands to aid troubleshooting... so use following as this toy container's Dockerfile

FROM python:3.5.2

CMD ["/bin/bash"]

so now issue this

docker build --tag saadi_now .   # creates image saadi_now

now launch that image

docker run -d  saadi_now  sleep infinity # launches container 

docker ps         #  lets say its container_id is b91f8cba6ed1

now login to that running container

docker exec -ti  b91f8cba6ed1 bash

cool so you are now inside the docker container so run the commands which were originally in the real Dockfile... this sometime makes it easier to troubleshoot

one by one add to this toy Dockerfile your actual commands from the real Dockerfile and redo above until you discover the underlying issues

Answer 2

Most likely this is related to either a bug in airflow with the werkzeug package, or your requirements might be clobbering something.

I recommend checking the versions of airflow, flask, and werkzueg that are used in the environment. It may be that you need to pin the version of flask or werkzueg as discussed here .