为什么 docker-compose python 没有找到气流操作员的模块

Question

I am working on tests for docker-airflow postgres etl. My project structure currently looks like this:

docker-airflow
|
├── Dockerfile
├── __init__.py
├── dags
│   ├── __init__.py
│   ├── pandas_etl.py
│   └── tuto.py
├── docker-compose.yml
├── operators
    ├── __init__.py
    └── pandas_etl_over_postgres_operator.py

When importing my pandas_etl_over_postgres_operator.py into the pandas_etl.py dag, I am getting an error that the module is not found.

The pandas_etl.py import code is:

from operators.pandas_etl_over_postgres_operator import PandasETLOverPostgresOperator

I have tried the following two alternatives, they also give the same error.

from .operators.pandas_etl_over_postgres_operator import PandasETLOverPostgresOperator

and

from ..operators.pandas_etl_over_postgres_operator import PandasETLOverPostgresOperator

The import works fine locally but fails when I build and run using docker-compose.

Answer 1

Please note that for airflow, by default, the [core] > dags_folder will have a value of /usr/local/airflow/dags meaning that airflow will look for dags at path /usr/local/airflow/dags .

As a result, all your dags code should be inside that folder and hence, here are a few things that you need to change for your code to work:

• In docker-compose.yml file:

- ./dags:/usr/local/airflow/dags/dags
- ./logs:/usr/local/airflow/dags/logs
- ./operators:/usr/local/airflow/dags/operators

• In pandas_etl.py file:

from operators.pandas_etl_over_postgres_operator import PandasETLOverPostgresOperator

Hope it helps!

Answer 2

I think it is because you need to put your operator inside a plugins directory and mount that into a container. Have a look at the Puckle documentation regarding plugins here . You can also see and change where your particular airflow instance looks for plugins by checking your configuration file here