Airflow 网络服务器无法识别 docker-compose 中的 Airflow 调度程序

Question

version: "3.7"
services:
    postgres:
        image: "postgres:13"
        container_name: "postgres"
        environment:
          - POSTGRES_USER=airflow
          - POSTGRES_PASSWORD=airflow
          - POSTGRES_DB=airflow
        ports:
        - "5431:5431"
        volumes:
        - ./data/postgres:/var/lib/postgresql/data


   webserver:
        image: jh111/airflow-hadoop-spark:test
        restart: always
        networks:
            - default_net
        environment:
            - AIRFLOW_HOME=/home/airflow
            - AIRFLOW__CORE__LOAD_EXAMPLES=False
            - AIRFLOW__CORE__dags_folder=/home/airflow/dags
        volumes:
            - ../dags:/home/airflow/dags #DAG folder
        ports:
            - "7777:7777"
        depends_on:
            - postgres
        command: bash -c "airflow webserver --port 7777"
    scheduler:
        image: jh111/airflow-hadoop-spark:test
        restart: always
        networks:
            - default_net
        environment:
            - AIRFLOW_HOME=/home/airflow
            - AIRFLOW__CORE__LOAD_EXAMPLES=False
            - AIRFLOW__CORE__dags_folder=/home/airflow/dags
            - AIRFLOW_WEBSERVER_HOST=webserver
        volumes:
            - ../dags:/home/airflow/dags #DAG folder
        depends_on:
            - postgres
        command: bash -c "airflow scheduler"

networks:
    default_net:

I checked that airflow webserver and scheduler is running on the server. However, when I get into webserver UI, I cannot find any dags in the web UI even if I have dag in the folder.

The scheduler does not appear to be running.

How can I solve this problem?

Answer 1

The default airflow executor is SequentialExecutor , with this executor the airflow scheduler and webserver should be running in the same host, which means the same docker image, and they use sqlite as a database (the postgres service which you created is not used, and AIRFLOW_WEBSERVER_HOST is not an airflow config).

To connect airflow to postgres server and to decouple airflow webserver and scheduler, you can use LocalExecutor , in this case you can run each component in a separate container, and they will interact with the database to communicate between each other.

You can do that by adding this two environment variables to the webserver and scheduler services:

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__EXECUTOR=LocalExecutor

You can check this project , it can help you to create and run a custom airflow image with LocalExecutor .