Airflow - 将 Xcom 拉取结果传递给 TriggerDagRunOperator conf

Question

Does anyone know what's the issue with the below code:

Essentially I am calling a TriggerDagRunOperator, and i am trying to pass some conf through to it, based off an XCOM Pull.

TRIGGER DAG:

def _should_trigger(**_):
    return {'Message': 'Hello World'}


should_trigger = PythonOperator(
    task_id="should_trigger",
    python_callable=_should_trigger,
    provide_context=True,
)

trigger_bar_dag = TriggerDagRunOperator(
    task_id="trigger_bar_dag",
    trigger_dag_id="bar",
    conf={"payload": "{{ task_instance.xcom_pull('should_trigger') }}"},
)

TARGET DAG:

@dag(dag_id="bar",
     default_args=default_args,
     schedule_interval=None
     )
def tasks():
    run_this = PythonOperator(
        task_id="run_this",
        python_callable=run_this_func,
        provide_context=True)

For some reason, in the run_this_func, i get payload: None. I can't seem to pass in values through the conf stream from an xcom pull. Does anyone know how this can be accomplished. I have also tried with different variations of xcom pull, like: ti.xcom_pull(key='return_value', task_ids=['should_trigger']) to no avail.

Thanks,

Answer 1

You need to specify should_trigger >> trigger_bar_dag because otherwise the XCom record may not yet exists and you will get:

[2021-06-06 08:23:35,898] {logging_mixin.py:104} INFO - {'payload': 'None'}

But once I add this relation then I'm getting:

[2021-06-06 08:21:41,356] {logging_mixin.py:104} INFO - {'payload': "{'Message': 'Hello World'}"}

with

def run_this_func(**context):
    print(context['params'])

Answer 2

here is my real code:

def external_trigger(name):

  def modify_dro(context, dagrun_order):
    log.info('context: {}'.format(context))
    # run_id from here
    log.info('dagrun_order: {}'.format(dagrun_order))
    dagrun_order.payload = {
      'branch': "branch123",
      'revision': 'revision123'
    }
    return dagrun_order

  exteranl_run = TriggerDagRunOperator(task_id='external_' + name,
                          trigger_dag_id='se_perf_post_test',
                          python_callable=modify_dro,
                          on_failure_callback=airflow_on_fail,
                          task_concurrency=256,
                          provide_context=True,
                          trigger_rule='all_done',
                          dag=dag)
  return exteranl_run

Use modify_dro func to pass variables for the triggered dag.

On the be triggered DAG:

def prepare_build(**args):
  log.info("args: " + str(args))
  run_id = args['run_id']
  log.info("run id " + run_id)
  conf = args['dag_run'].conf
  log.info('conf: {}'.format(conf))
  branch = conf.get('branch')
  revision = conf.get('revision')
  log.info("run build on branch: {}, revision: {}".format(branch, revision))

Hope it can help.