Is it possible to pass user input from dag to sub dag in airflow?

Question

I am trying to generate dynamic workflow in airflow based on user input. I know there is option to have it based on data from file and database but in all these cases, workflow will not directly be dependent on user input and in case where multiple users are using same dag then in that case also issue may come. To avoid all these, i am thinking of passing user input to sub dag and generate the workflow. But subdag does not have option of passing user input from ui.

Answer 1

you can use Variable in Airflow, according to documentation

Variables are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. Variables can be listed, created, updated and deleted from the UI (Admin -> Variables), code or CLI.

you can refer for the following link for further understanding:

• https://www.applydatascience.com/airflow/airflow-variables/

Answer 2

You can use xcom to pass user input in Airflow => custom-xcom-backend documentation

Using airflow-rest-plugin , fetch the input and pass in internally via xcom to subdag.

Answer 3

There are many tricks for getting the same done, but the actual solution should come from airflow by way of dynamic task which is missing at present. Hopefully we will see that in future version of airflow.

Answer 4

I guess using variables is a good solution for a problem BUT users may overwrite each other changes (some issues can occur).

Alternative 1:

Airflow has a REST API on top which supports dag triggering functionality.

Request example:

curl -X POST \
'http://localhost:8080/api/experimental/dags/<DAG_ID>/dag_runs' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--data '{"conf":"{\"key\":\"value\"}"}'

The data section can store some user input which will later be accessed in Airflow operators.

More documentation: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_import_error

Alternative 2:

Airflow supports a CLI interface that can be used for triggering dags. You can specify extra configurations as a configuration parameter ( -c option). Configurations can store user input.

Comand format:

airflow trigger_dag [-h] [-sd SUBDIR] [-r RUN_ID] [-c CONF] [-e EXEC_DATE]
                    dag_id

More documentation: http://airflow.apache.org/docs/apache-airflow/1.10.5/cli.html#trigger_dag

StackOverflow question which shows how configuration parameters can be accessed in Airflow operators: Accessing configuration parameters passed to Airflow through CLI