Airflow metrics with prometheus and grafana

Question

any one knows how to send metrics from airflow to prometheus, I'm not finding much documents about it, I tried the airflow operator metrics on Grafana but it doesnt show any metrics and all it says no data points.

Answer 1

By default, Airflow doesn't have any support for Prometheus metrics. There are two ways I can think of to get metrics in Prometheus.

1. Enable StatsD metrics and then export it to Prometheus using statsd exporter .

2. Install third-party/open-source Prometheus exporter agents (ex. airflow-exporter ).

If you are going with 2nd approach then the Airflow Helm Chart also provides support for that.

Edit

If you're using statsd exporter here is a good resource for Grafana Dashboard and exporter config.

Answer 2

This is how it worked for me -

• Running airflow in docker using this doc
• Added this configuration inside the docker-compose file downloaded in the previous step AIRFLOW__SCHEDULER__STATSD_ON: 'true'
AIRFLOW__SCHEDULER__STATSD_HOST: statsd-exporter
AIRFLOW__SCHEDULER__STATSD_PORT: 9125
AIRFLOW__SCHEDULER__STATSD_PREFIX: airflow
  Under environment section
• Now run the statsd_export
• docker run -d -p 9102:9102 -p 9125:9125 -p 9125:9125/udp \ -v $PWD/statsd_mapping.yml:/tmp/statsd_mapping.yml \ prom/statsd-exporter --statsd.mapping-config=/tmp/statsd_mapping.yml
• Get the statsd_mapping.yml contents from Here
• Now do docker-compose up to run the airflow and try to run some worflow and you should see logs at http://localhost:9102/metrics

Answer 3

If you installed your Airflow with statsd support:

pip install 'apache-airflow[statsd]'

you can expose Airflow statsd metrics in the scheduler section of your airflow.cfg file, something like this:

[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

Then, you can install a tool called statsd_exporter , that captures statsd-format metrics and converts them to Prometheus-format, making them available at the /metrics endpoint for Prometheus to scrape.

There is a docker image available on DockerHub called astronomerinc/ap-statsd-exporter that already maps Airflow statsd metrics to Prometheus metrics.

References:

• https://airflow.apache.org/docs/stable/metrics.html
• https://github.com/prometheus/statsd_exporter
• https://hub.docker.com/r/astronomerinc/ap-statsd-exporter/tags

Answer 4

Have you tried using roobinhood's airflow-prometheus-exporter ? Sounds promising but I haven't tried yet.

Answer 5

Nice article from Redhat, I think it will be useful https://www.redhat.com/en/blog/monitoring-apache-airflow-using-prometheus Editing the config airflow.cfg #./statsd_exporter --statsd.listen-udp localhost:8125 --log.level debug