如何从气流中的任务动态生成下游列表

Question

I have a main task holding it's logic in this function

I'm not completely sure how to do this. Maybe I need another task in between? Any help is appreciated. Thanks!

Answer 1

If I understood correctly, you already have multiple tasks created, but you need to dynamically define which of them will follow downstream execution. If that is the case you can safely use BranchPythonOperator :

It derives the PythonOperator and expects a Python function that returns a single task_id or list of task_ids to follow. The task_id(s) returned should point to a task directly downstream from {self}. All other "branches" or directly downstream tasks are marked with a state of skipped so that these paths can't move forward. The skipped states are propagated downstream to allow for the DAG state to fill up and the DAG run's state to be inferred.

Consider the following based on the example_dag distributed with Airflow:

with DAG(
    dag_id="branch_multiple_tasks",
    default_args=args,
    start_date=days_ago(1),
    schedule_interval="@daily",
    tags=["example"],
) as dag:

    run_this_first = DummyOperator(
        task_id="run_this_first",
    )

    options = ["branch_a", "branch_b", "branch_c", "branch_d"]

    branching = BranchPythonOperator(
        task_id="branching",
        python_callable=lambda: options[1:3],
    )
    run_this_first >> branching

    join = DummyOperator(
        task_id="join",
        trigger_rule="none_failed_or_skipped",
    )

    for option in options:
        t = DummyOperator(
            task_id=option,
        )

        dummy_follow = DummyOperator(
            task_id="follow_" + option,
        )

        # Label is optional here, but it can help identify more complex branches
        branching >> Label(option) >> t >> dummy_follow >> join

In this example the python_callable passed to branching task is hardcoded to return ['branch_b', 'branch_c'] . You could provide your own callable and return a list of tasks_id 's as string, based on any criteria. You could even use your get_campaign_active function as long as you return the expected format. Maybe it's cleaner if you create a new function and perform xcom_pull from the previous one. I guess it's up to your needs.

Graph view:

Let me know if that worked for you!