Prometheus alertmanager 向多个 slack 通道发送通知

Question

We have two different teams working on different applications.I would like send alert notifications to different slack channels via using same alerts expressions. I found some examples but not understand what is the main reason to use receiver: 'default' when try to add new route? What is the role of this and what if it affects if ı change this?

Meanwhile will be appreciate if you can help how should I send the notifations to multiple slack channels.. New one is what I tried.

Current alertmanager.yml

receivers:
  - name: 'team-1'
    slack_configs:
    - api_url: 'https://hooks.slack.com/services/1'
      channel: '#hub-alerts'
route:
  group_wait: 10s
  group_interval: 5m
  receiver: 'team-1'
  repeat_interval: 1h
  group_by: [datacenter]

---

New alertmanager.yml

alertmanager.yml:
    receivers:
      - name: 'team-1'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/1'
          channel: '#channel-1'
          send_resolved: true
      
      - name: 'team-2'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/2'
          channel: '#channel-2'
          send_resolved: true

route:
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h
  group_by: [datacenter]
  receiver: 'default'
  routes:
  - receiver: 'team-1'
  - receiver: 'team-2'

---

Answer 1

You need to set the continue property on your route to true. By default it is false.

The default behaviour of AlertManager is to traverse your routes for a match and exit at the first node it finds a match at.

What you want to do is fire an alert at the match and continue to search for other matches and fire those too.

Relevant documentation section:https://prometheus.io/docs/alerting/latest/configuration/#route

An example using this: https://awesome-prometheus-alerts.grep.to/alertmanager.html

In-lined the example above in case it ever breaks.

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "slack"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "slack"
      group_wait: 10s
      match_re:
        severity: critical|warning
      continue: true

    - receiver: "pager"
      group_wait: 10s
      match_re:
        severity: critical
      continue: true

receivers:
  - name: "slack"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxx'
        send_resolved: true
        channel: 'monitoring'
        text: "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

  - name: "pager"
    webhook_config:
      - url: http://a.b.c.d:8080/send/sms
        send_resolved: true