从 Airflow 中的另一个 DAG 获取 DAG 电子邮件

Question

I want to get the email mentioned in this DAG's default args using another DAG in the airflow. How can I do that? Please help, I am new to airflow!

from airflow.models import DagRun
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
from datetime import datetime, timedelta
from airflow import DAG

def first_function(**context):
    print("hello")

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['example@gmail.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

with DAG(
    'main',
    default_args=default_args,
    description='Sample DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2022,6,10),
    catchup=False
) as dag:
    first_function = PythonOperator(
        task_id="first_function",
        python_callable=first_function,
    )
first_function

Answer 1

You can use a custom module in Airflow to share config options/Operators/or any arbitrary Python code across DAGs.

Typically you would create a directory in your DAGs directory (which by default is {AIRFLOW_HOME}/dags .

To share default_args between DAGs, you could create the following layout:

• Create {AIRFLOW_HOME}/dags/custom/__init__.py
• Create {AIRFLOW_HOME}/dags/custom/shared_config.py
• Create {AIRFLOW_HOME}/dags/.airflowignore
• Add the directory name custom to the first line of .airflowignore .
• Cut and paste your default_args dictionary from your DAG into {AIRFLOW_HOME}/dags/custom/shared_config.py

You can see this layout suggested in the Airflow documentation here .

The .airflowignore tells the scheduler to skip the custom directory when it parses your DAGs (which by default happens every 30s) - because the custom directory contains Python, but never any DAGs, the scheduler should skip these files to avoid adding unnecessary load to the scheduler. This is explained in the documentation link above.

You need to add an __init__.py to the custom directory - airflow requires it even though when writing in Python3 you don't need it because of implicit namespaces (again this is explained in the same link above).

From your dag you can then import as needed:

from custom.shared_config import default_args