无法在 Airflow 1.10.14 上运行 DAG，在官方 Apache\Airflow 图像上运行 docker-compose

Question

I've been trying to set Airflow 1.10.14 to execute Python based process, on docker container using docker-compose. The host is Ubuntu 18 VM.

My Dockerfile:

FROM apache/airflow:1.10.14
USER root
RUN pip install --upgrade pip
RUN pip install --user psycopg2-binary

COPY airflow.cfg /opt/airflow/

RUN apt-get update && apt-get install -y \
    libodbc1 \
    python3-dev\
    libevent-dev\
    unixodbc-dev \
    freetds-dev \
    freetds-bin -y \ 
    tdsodbc -y \
    build-essential

# install dependencies
ADD requirements.txt .
RUN pip install -r requirements.txt


USER airflow

Then I execute:

docker build -t learning/airflow .

And my docker-compose.yml is:

version: "3"
networks:
  airflow:

services:
  postgres:
    image: "postgres:9.6"
    container_name: "postgres"
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
      - "5432:5432"
    networks:
      - airflow

  # uncomment initdb if you need initdb at first run
  initdb:
    image: learning/airflow
    entrypoint: airflow db init
    depends_on:
      - postgres
    networks:
      - airflow

  webserver:
    image: learning/airflow
    restart: always
    depends_on:
      - postgres
    volumes:
      - ./dags:/opt/airflow/dags
    ports:
      - "8080:8080"
    entrypoint: airflow webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3
    networks:
      - airflow

  scheduler:
    image: learning/airflow
    restart: always
    depends_on:
      - postgres
      - webserver
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
    entrypoint: airflow scheduler
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-scheduler.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3
    networks:
      - airflow

I also use the airflow.cfg as appear in here (with minor changes) In the first run I execute in 3 steps in separated terminals:

docker-compose up postgres
docker-compose up initdb
docker-compose up webserver scheduler

I'm able to access the Airflow UI and turn on the DAG but first step fails immediately with the following error:

*** Log file does not exist: /opt/airflow/logs/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
*** Fetching from: http://bf23abdeb4b0:8793/log/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='bf23abdeb4b0', port=8793): Max retries
exceeded with url:
/log/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection
object at 0x7ff6f20ae898>: Failed to establish a new connection:
[Errno 111] Connection refused',))

What am I missing here? any help will be appreciated...

Answer 1

Try resetting the metadata db and then build again?

airflow resetdb

Answer 2

I may be late, but I think the problem is on this line on your Dockerfile:

COPY airflow.cfg /opt/airflow/

Should be like this:

COPY airflow.cfg /opt/airflow/airflow.cfg