Dynamic task in airflow upon REST parameter

Question

I trigger an airflow DAG and pass REST parameters. Upon a REST parameter list, I want to repeat some of the tasks in this DAG. After some tries I got stuck and I am not sure if this is possible. Here one try:

def determine_rest_params(**kwargs):
    values_comma_sep = kwargs["dag_run"].conf["myparam"]
    values= []
    if values_comma_sep :
        values= values_comma_sep .split(",")
    return values


def create_task_for_param(p, **kwargs)
    # create an operator instance

with airflow.DAG("get_prediction2", default_args=default_args, schedule_interval=None) as dag:
    
    start = DummyOperator(
        task_id='start',
        dag=dag
     )

    params = determine_rest_params()
    for cur_p in params:
          cur_task = create_task_for_param(cur_p)
          start >> cur_task

I only see the start task and no other operator. Is it possible in general? Regards Oli

Answer 1

You can try this: (Not sure if this will work) Change the for loop to:

for i in range(0, len(params):
    params[i] = create_task_for_param(params[i])
    if i == 0:
        start.set_downstream(params[i])
    else:
        params[i-1].set_downstream(parmas[i])

I am also not sure if you are getting the params right way. Just print it and see if you getting params. if yes then the above for loop should work. if not you can try getting params in a start task.