Prometheus Operator Alertmanager - 自定义规则 - 在 config.plain 类型中找不到字段组

Question

我已经通过 helm 安装了 prometheus-operator，现在想要设置自定义警报规则，设置了电子邮件通知，目前我正在收到每个通知，我想“使其静音”，以便我可以收到自定义警报的电子邮件。

警报管理器.yaml：

global:
  resolve_timeout: 5m
route:
  receiver: 'email-alert'
  group_by: ['job']


  routes:
  - receiver: 'email-alert'
    match:
      alertname: etcdInsufficientMembers
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h  

receivers:
- name: email-alert
  email_configs:
  - to: receiver@example.com
    from: sender@example.com
    # Your smtp server address
    smarthost: smtp.office365.com:587
    auth_username: sender@example.com
    auth_identity: sender@example.com
    auth_password: pass

以上文件应用成功，

我加入以下在上述文件的末尾线，作为参考这里：

# Example group with one alert

groups:
- name: example-alert
  rules:
    # Alert about restarts
  - alert: RestartAlerts
    expr: count(kube_pod_container_status_restarts_total) > 0
    for: 1s
    annotations:
      summary: "More than 5 restarts in pod {{ $labels.pod-name }}"
      description: "{{ $labels.container-name }} restarted (current value: {{ $value }}s) times in pod {{ $labels.pod-namespace }}/{{ $labels.pod-name }}

然后在 pod 日志中，我得到了这个：

="Loading configuration file failed" file=/etc/alertmanager/config/alertmanager.yaml err="yaml: unmarshal errors:\n  line 28: field groups not found in type config.plain"

Answer 1

解决了，首先需要列出所有可用的规则：

    kubectl -n monitoring get prometheusrules
NAME                                                              AGE
prometheus-prometheus-oper-alertmanager.rules                     29h
prometheus-prometheus-oper-etcd                                   29h
prometheus-prometheus-oper-general.rules                          29h
prometheus-prometheus-oper-k8s.rules                              29h
prometheus-prometheus-oper-kube-apiserver-error                   29h
prometheus-prometheus-oper-kube-apiserver.rules                   29h
prometheus-prometheus-oper-kube-prometheus-node-recording.rules   29h
prometheus-prometheus-oper-kube-scheduler.rules                   29h
prometheus-prometheus-oper-kubernetes-absent                      29h
prometheus-prometheus-oper-kubernetes-apps                        29h
prometheus-prometheus-oper-kubernetes-resources                   29h
prometheus-prometheus-oper-kubernetes-storage                     29h
prometheus-prometheus-oper-kubernetes-system                      29h
prometheus-prometheus-oper-kubernetes-system-apiserver            29h
prometheus-prometheus-oper-kubernetes-system-controller-manager   29h
prometheus-prometheus-oper-kubernetes-system-kubelet              29h
prometheus-prometheus-oper-kubernetes-system-scheduler            29h
prometheus-prometheus-oper-node-exporter                          29h
prometheus-prometheus-oper-node-exporter.rules                    29h
prometheus-prometheus-oper-node-network                           29h
prometheus-prometheus-oper-node-time                              29h
prometheus-prometheus-oper-node.rules                             29h
prometheus-prometheus-oper-prometheus                             29h
prometheus-prometheus-oper-prometheus-operator                    29h

然后选择一个进行编辑，或者删除除默认的之外的所有： prometheus-prometheus-oper-general.rules

我选择编辑节点出口商规则

kubectl edit prometheusrule prometheus-prometheus-oper-node-exporter -n monitoring

在文件末尾添加了这些行

- alert: RestartAlerts
  annotations:
    description: '{{ $labels.container }} restarted (current value: {{ $value}}s)
          times in pod {{ $labels.namespace }}/{{ $labels.pod }}'
    summary: More than 5 restarts in pod {{ $labels.container }}
  expr: kube_pod_container_status_restarts_total{container="coredns"} > 5
  for: 1min
  labels:
    severity: warning

不久之后，我收到了有关此警报的电子邮件。