使用从 POST Json 传递的 Kwargs 定义全局 Airflow 变量

Question

I'm creating a DAG and that needs functionality to set global variables using kwargs passed in from the POST Json used to trigger the job. So far, I have attempted this way:

import airflow
from airflow import DAG
from datetime import timedelta

DAG_Name = 'dag_test'

DEFAULT_ARGS = {
    'owner': '...',
    'depends_on_past': False,
    'email': ['...'],
    'email_on_failure': True,
    'start_date': datetime(2020,8,31)
}

dag = DAG(DAG_Name, default_args=DEFAULT_ARGS, dagrun_timeout=timedelta(hours=2))

snap_date = ''
output_loc = ''
recast = ''

def define_param(**kwargs):
    global snap_date
    global output_loc
    global recast
    snapshot = str(kwargs['dag_run'].conf['snap_date'])
    output_s3 = kwargs['dag_run'].conf['output_loc']
    recast = str(kwargs['dag_run'].conf['recast'])


DEFINE_PARAMETERS = PythonOperator(
    task_id='DEFINE_PARAMETERS',
    python_callable=define_param,
    provide_context=True,
    dag=dag)

But this does not work. How would I use kwargs to set global dag variables?

Answer 1

Use Variable.set as it will make the actual update to the database, along with handling session and serialization for you if needed.

Variable.set("snap_date", "2019-09-17")

Ref: https://github.com/apache/airflow/blob/1.10.1/airflow/models.py#L4558-L4569