Get Airflow Dag Concurrent Runs in MWAA

Question

Can we get the airflow dag run count( concurrency), which runs in MWAA? I set the concurrency on Dag Level. But I would like to get dag concurrency, and if it is below a specific limit would like to trigger the dag.

Since the dag is running in MWAA, is there a metric available to get this count? I found PoolRunningSlots metric which is very close to get Concurrent Dag run count( checked couple of times, this value is same as the run count of the dag ). FYI, PoolRunningSlots https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html#access-metrics-cw-console-v202

Please let me know if you encounter this before. TIA.

Answer 1

I don't know why do you want to do it manually, because Airflow can do it by just configuring max_active_runs (use your specific limit) which defines how many running concurrent instances of a DAG there are allowed to be.

For PoolRunningSlots metric, it's the number of slots used for a specific pool, so if you have a pool used in different dags, or different pools used in your dag (a different pool for each task), or the tasks used different number of slots (check the doc ), in these cases PoolRunningSlots will be very different from the concurrent Dag runs count.

If you really need this metric, you can create a custom metric by adding a task at the beginning of your dag which increment a cloud watch metric ( doc ), and adding on_success_callback and on_failure_callback to your dag to decrease the same metric, in this case this new metric will represent the count of the active runs of your dag.