Dynamically generate a DAG during a DAG Run - Airflow

Question

I'd like to know if you can easily, dynamically generate a DAG within a DAG run, like:

DAG A would be:

1. Do some data computations
2. Based on the previous result generate DAG B
3. Load DAG B into the airflow DagBag
4. TriggerDagRunOperator for DAG B
5. ExternalTaskSensor on last DAG B task.

Cheers,

Answer 1

This is possible. You would need to generate a file that, when parsed generates DAG B. You could not just generate a DAG object as part of DAG A. You will need write access to the DagBag folder across all your Airflow components (scheduler, webserver, and workers). Also, allow some time for the DAG B file to be parsed before triggering it with TriggerDagRun Operator.

All of this is a little hacky. AIP-42 is being worked on that will allow for modifying the tasks within a DAG based on other results which I think will help you get the desired effect.