將指標從 telegraf 發送到 prometheus

Question

我在同一台主機上運行prometheus和telegraf 。

我正在使用一些輸入插件：

• 輸入.cpu
• 輸入.ntpq

我已經配置了prometheus_client output 插件來發送數據給 prometheus

這是我的配置：

    [[outputs.prometheus_client]]
      ## Address to listen on.
      listen = ":9126"

      ## Use HTTP Basic Authentication.
      # basic_username = "Foo"
      # basic_password = "Bar"

      ## If set, the IP Ranges which are allowed to access metrics.
      ##   ex: ip_range = ["192.168.0.0/24", "192.168.1.0/30"]
      # ip_range = []

      ## Path to publish the metrics on.
      path = "/metrics"

      ## Expiration interval for each metric. 0 == no expiration
      #expiration_interval = "0s"

      ## Collectors to enable, valid entries are "gocollector" and "process".
      ## If unset, both are enabled.
      # collectors_exclude = ["gocollector", "process"]

      ## Send string metrics as Prometheus labels.
      ## Unless set to false all string metrics will be sent as labels.
      # string_as_label = true

      ## If set, enable TLS with the given certificate.
      # tls_cert = "/etc/ssl/telegraf.crt"
      # tls_key = "/etc/ssl/telegraf.key"

      ## Export metric collection time.
      #export_timestamp = true

Here's my prometheus config

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

#  - job_name: 'node_exporter'
#    scrape_interval: 5s
#    static_configs:
#      - targets: ['localhost:9100']

  - job_name: 'telegraf'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9126']

如果我要去http://localhost:9090/metrics我看不到任何來自 telegraf 的指標。

我也從 telegraf 捕獲了一些日志

/opt telegraf --config /etc/telegraf/telegraf.conf --input-filter filestat --test ➜ /opt tail -F /var/log/telegraf/telegraf.log 2019-02-11T17:34:20Z D. [outputs.prometheus_client] wrote batch of 28 metrics in 1:234869ms 2019-02-11T17:34.20ZD: [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics: 2019-02-11T17.34.30ZD: [outputs:file] wrote batch of 28 metrics in 384.672µs 2019-02-11T17:34.30ZD: [outputs:file] buffer fullness. 0 / 10000 metrics. 2019-02-11T17:34:30Z D. [outputs:prometheus_client] wrote batch of 30 metrics in 1.250605ms 2019-02-11T17:34:30Z D! [outputs.prometheus_client] buffer fullness: 9 / 10000 metrics.

我沒有在日志中看到問題。

Answer 1

Prometheus 服務器的/metrics端點導出有關服務器本身的指標，而不是它從 telgraf 導出器等目標中抓取的指標。

轉到http://localhost:9090/targets ，您應該會看到 Prometheus 服務器正在抓取的目標列表。 如果配置正確，telegraf 導出器應該是其中之一。

要查詢 Prometheus 以獲取 Telegraf 導出器生成的指標，請將瀏覽器導航到http://localhost:9090/graph並在查詢字段中輸入例如cpu_time_user 。 如果啟用了 CPU 插件，它應該有更多的指標。

Answer 2

您應該使用以下 Prometheus 配置文件來抓取 Telegraf 的prometheus_client導出的指標：

scrape_configs:
- job_name: telegraf
  static_configs:
  - targets:
    - "localhost:9126"

啟動 Prometheus 時，必須將此文件的路徑傳遞給--config.file命令行標志。

在這些文檔中查看有關 Prometheus 配置的更多詳細信息。

PS 有一種替代解決方案可以將 Telegraf 收集的指標直接推送到類似 Prometheus 的系統，例如VictoriaMetrics而不是 InfluxDB - 請參閱這些文檔。 稍后可以使用 PromQL 兼容的查詢語言 - MetricsQL查詢這些指標。