Collecting metrics from multiple telegraf to prometheus

Question

Continue on from the question of Sending metrics from telegraf to prometheus , which covers the case of single telegraf agent, what's the suggested setup to collect metrics from multiple telegraf to prometheus?

In the end, I want prometheus to chart (on the same graph), CPU usage of server-1, server-2, ... to server-n, in their own lines.

Answer 1

Taking the configuration from the original post, you can simply add targets to you telegraf job; supposing that the same Telegraf config is used on each server.

scrape_configs:
  - job_name: 'telegraf'
    scrape_interval: 5s
    static_configs:
      - targets: ['server-1:9126','server-2:9126',...]

It will produce the metrics (ex: cpu_time_user) with different instance tag corresponding to the targets configured. Typing the metric name in Prometheus will display all of them.

If you really want to see only the name of the server, you can use metric_relabel_configs to generate an additional label:

scrape_configs:
  - job_name: 'telegraf'
    ...
    metric_relabel_configs:
    - source_labels: [instance]
      regex: '(.*):\d+'
      target_label: server

Automatically adding servers to your Prometheus config is a matter of service discovery.