How configure a "global" prometheus for services running multiple docker network

Question

I have a "global" network where Prometheus is running, then there are multiple mini networks where different microservices run. I'm trying to scrape the metrics from the microservices (under those multiple mini networks) without adding all the mini-networks to the "global" network.

I checked How to configure Prometheus in a multi-location scenario? but I don't think it's the right way for my scenario.

Answer 1

Running on Docker Swarm we can take advantage of Docker Swarms mesh networking to achieve this.

First, you need some things - a master prometheus stack that defines a global network that this prometheus is attached to.

master-prometheus-stack.yml

networks:
  prometheus:
    name: prometheus
    driver: overlay
    attachable: true

configs:
  prometheus-1:
    file: prometheus-master.yml

services:
  prometheus:
    image: prom/prometheus
    networks:
    - prometheus
    configs:
      source: prometheus-1
      target: /etc/prometheus/prometheus.yml

This prometheus instance is configured with a single job: Its looking for the dnsrr record tasks.prometheus.scrape for child prometheus instances, and scraping everything from their federate entrypoint.

prometheus-master.yml

scrape_configs:
  - job_name: federate-prometheus
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~".*"}'
    dns_sd_configs:
    - names: [ tasks.prometheus.scrape ]
      type: 'A'
      port: 9090

Then, the next part of this cunning plan is simple: Each microservice stack deploys with its own private prometheus instance.

stack.yml

networks:
  stack:
  prometheus_public:
    name: prometheus
    external: true

services:
   other-services:
     ...

   prometheus:
     image: prom/prometheus:
     networks:
       stack:
       prometheus_public:
         aliases: ["prometheus.scrape"]
       configs:
       - source: prometheus-1
         target: /etc/prometheus/prometheus.yml

This prometheus instance should have a prometheus config file with all the scrape jobs required for each microservice in the stack. The prometheus instance is attached to the microservice network so it can reach all the microservices to scrape them. It is also attached to the public prometheus network, isolating the rest of the stack from this connection. To advertize the fact that it wants to be scraped, we add a specific alias on that network "prometheus.scrape", which ensures that docker swarm adds the ip for this prometheus to the dnsrr record tasks.prometheus.scrape on that network.

Now, because dns_sd_configs is dynamically probed, as microservices, each with their own prometheus instance, are deployed they add their own "prometheus.scrape" aliases to the common network, and the main prometheus instance will discover them and incorporate their metrics from their /federate endpoints automatically. Conversely, removing a stack cleanly de-registers the extra instances.

Separating the jobs top actually generate discreet dashboards, perhaps by adding extra tags identifying the source stack, remain an exercise for the reader.