Prometheus and Alertmanager - route based on env label

Question

I'm trying to configure alertmanager so that it sends alerts to the right channels, based on value of a specific label. I have 3 slack channels - dev/staging/prod and I want the alerts coming from instances that have "env" label set to dev to be sent to the dev slack channel. Staging and prod would obviously work in the same manner. Here is part of my config:

global:
  resolve_timeout: 1m
  slack_api_url: 'https://slack-url'

route:
  group_by: [...]
  receiver: 'default'
  routes:
  - match:
      env: 'prod'
    receiver: 'slack-notifications-prod'
  - match:
      env: 'staging'
    receiver: 'slack-notifications-staging'
  - match:
      env: 'dev'
    receiver: 'slack-notifications-dev'

receivers:
- name: 'default'
- name: 'slack-notifications-prod'
...
- name: 'slack-notifications-staging'
...
- name: 'slack-notifications-dev'
...

The slack-notifications receivers are all the same and they only differ in one thing, which is the appropriate channel name.

Current behaviour: All alerts are sent to the prod slack channel

Expected behaviour: Alerts from "dev" env are sent to dev channel, "staging" to staging channel, and "prod" to prod channel.

Alertmanager sees these labels just fine (judging from the info from alertmanager webUI).

Answer 1

Turns out my config was fine and I was using a webhook URL which was tied only to one slack channel, I wasn't aware of that.

Answer 2

You have to add continue: true attribute on the first match:

global:
  resolve_timeout: 1m
  slack_api_url: 'https://slack-url'

route:
  group_by: [...]
  receiver: 'default'
  routes:
  - match:
      env: 'prod'
    receiver: 'slack-notifications-prod'
    continue: true
  - match:
      env: 'staging'
    receiver: 'slack-notifications-staging'
  - match:
      env: 'dev'
    receiver: 'slack-notifications-dev'

receivers:
- name: 'default'
- name: 'slack-notifications-prod'
...
- name: 'slack-notifications-staging'
...
- name: 'slack-notifications-dev'
...

The AlertManager will evaluate children routes until there are no routes left or no routes for a given level are matching the current alert.

In that case, the AlertManager will take the configuration of the current node evaluated.

The continue attribute is a value used to define if you want to evaluate route siblings (belonging to the same level) if a route on the same level was already matching.

https://devconnected.com/alertmanager-and-prometheus-complete-setup-on-linux/