使用 prometheus 和 alertmanager 時警報消息未顯示在 slack 中

Question

我正在嘗試使用alertmanager 使Prometheus 發現的警報在 slack 中得到通知。

這是alert.rules文件並且工作正常

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

它正在成功顯示一個實例。

但是我的alertmanager.yml有什么問題，它沒有將通知發送到 slack。 我還成功設置了 slack webhook，甚至在使用 slack 提供的服務創建鈎子時測試了鈎子是否工作正常

警報管理器.yml

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'
[tgurung@ip131 prometheus_graphana_myversion]$ cat alertmanager/alertmanager.yml 


route: 
  receiver: 'slack-notifications'
  #group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: https://hooks.slack.com/services/T52GRFN3F/B93KTCUHH/JC
    channel: #general
    send_resolved: true

    # Alertmanager templates apply here.
    text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

運行 docker-compose up 時，我得到以下信息

prometheus_1     | level=error ts=2018-02-06T09:36:35.580565429Z caller=notifier.go:454 component=notifier alertmanager=http://x.x.x.x:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://x.X.x.x:9093/api/v1/alerts: dial tcp x.x.x.x:9093: getsockopt: no route to host"

解決上述錯誤：為了解決上述路由問題，我在全新的實例中運行了警報管理器，然后克服了該錯誤

轉到錯誤消息中的 API 鏈接，我可以看到這一點

{"status":"success","data":[]}

這是 alert_manager 的，看起來很好用。

alertmanager_1   | level=info ts=2018-02-06T09:36:37.66654544Z caller=main.go:141 msg="Starting Alertmanager" version="(version=0.13.0, branch=HEAD, revision=fb713f6d8239b57c646cae30f78e8b4b8861a1aa)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.66661402Z caller=main.go:142 build_context="(go=go1.9.2, user=root@d83981af1d3d, date=20180112-10:32:46)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.668103448Z caller=main.go:279 msg="Loading configuration file" file=/alertmanager/alertmanager.yml
alertmanager_1   | level=info ts=2018-02-06T09:36:37.673288146Z caller=main.go:354 msg=Listening address=:9093

這是 prometheus.yml 配置文件

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

#alerting rules file
rule_files:
  - '/alertmanager/alert.rules'


scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
       - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']


alerting:
  alertmanagers:
    - static_configs:
      - targets: ["54.36.X.X:9093"]  #this is the alertmanager service url 

這是我的 docker-compose.yml

version: '2'
volumes:
    grafana_data: {}

services:
    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanager/alert.rules:/alertmanager/alert.rules
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'
        links:
            - "alertmanager"


    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/alertmanager/alertmanager.yml'
        ports:
            - '9093:9093'

警報管理器狀態鏈接不顯示從 docker-composer 中的卷傳遞的配置。 它顯示默認配置

Answer 1

正如一些人已經指出的那樣，alertmanager 配置只包括如何發送警報而不是創建警報。 那是普羅米修斯的工作。 看看這個 repo，它非常簡單地將 prometheus 設置為 alertmanager

https://github.com/stefanprodan/dockprom