[英]How to trigger alert in Kubernetes using Prometheus Alert Manager
我在集群中安装了kube-prometheus( https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus )。 它包含一些默认警报,例如“ CoreDNSdown等”。 如何创建我自己的警报?
有人可以提供示例示例来创建警报,该警报将向我的gmail帐户发送电子邮件吗?
当Docker容器容器位于Error或CarshLoopBackOff kubernetes中时,我遵循此警报 。 但是我无法使它工作。
要将警报发送到您的gmail帐户,您需要在一个名为alertmanager.yaml的文件中设置alertmanager配置:
cat <<EOF > alertmanager.yml
route:
group_by: [Alertname]
# Send all notifications to me.
receiver: email-me
receivers:
- name: email-me
email_configs:
- to: $GMAIL_ACCOUNT
from: $GMAIL_ACCOUNT
smarthost: smtp.gmail.com:587
auth_username: "$GMAIL_ACCOUNT"
auth_identity: "$GMAIL_ACCOUNT"
auth_password: "$GMAIL_AUTH_TOKEN"
EOF
现在,当你使用KUBE-普罗米修斯所以你将有一个名为秘密alertmanager-main
是对默认配置alertmanager
。 您需要使用以下命令使用新配置再次创建一个secret alertmanager-main
:
kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring
现在,您可以将alertmanager设置为在收到来自普罗米修斯的警报时发送电子邮件。
现在,您需要设置一个警报,以发送邮件。 您可以设置DeadManSwitch警报,该警报在每种情况下都会触发,并用于检查警报管道
groups:
- name: meta
rules:
- alert: DeadMansSwitch
expr: vector(1)
labels:
severity: critical
annotations:
description: This is a DeadMansSwitch meant to ensure that the entire Alerting
pipeline is functional.
summary: Alerting DeadMansSwitch
之后,将触发DeadManSwitch
警报,并应将电子邮件发送到您的邮件中。
参考链接:
编辑:
Deadmanswitch警报应进入您的普罗米修斯正在读取的配置映射中。 我将在这里分享我的普罗米修斯的相关快照:
"spec": {
"alerting": {
"alertmanagers": [
{
"name": "alertmanager-main",
"namespace": "monitoring",
"port": "web"
}
]
},
"baseImage": "quay.io/prometheus/prometheus",
"replicas": 2,
"resources": {
"requests": {
"memory": "400Mi"
}
},
"ruleSelector": {
"matchLabels": {
"prometheus": "prafull",
"role": "alert-rules"
}
},
上面的配置是我的prometheus.json文件的名称,该文件具有要使用的ruleSelector
的名称,以及ruleSelector
,它将基于prometheus
和role
标签选择规则。 所以我的规则配置映射如下:
kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-rules
namespace: monitoring
labels:
role: alert-rules
prometheus: prafull
data:
alert-rules.yaml: |+
groups:
- name: alerting_rules
rules:
- alert: LoadAverage15m
expr: node_load15 >= 0.50
labels:
severity: major
annotations:
summary: "Instance {{ $labels.instance }} - high load average"
description: "{{ $labels.instance }} (measured by {{ $labels.job }}) has high load average ({{ $value }}) over 15 minutes."
在上面的配置图中替换DeadManSwitch
。
如果您使用的是kube-promehtheus,则默认情况下它具有alertmanager-main secret和prometheus kind设置。
步骤1:您必须删除alertmanager-main机密
kubectl delete secret alertmanager-main -n monitoring
第2步:如Praful所述,用新的变化创造秘密
cat <<EOF > alertmanager.yaml
route:
group_by: [Alertname]
# Send all notifications to me.
receiver: email-me
receivers:
- name: email-me
email_configs:
- to: $GMAIL_ACCOUNT
from: $GMAIL_ACCOUNT
smarthost: smtp.gmail.com:587
auth_username: "$GMAIL_ACCOUNT"
auth_identity: "$GMAIL_ACCOUNT"
auth_password: "$GMAIL_AUTH_TOKEN"
EOF
kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring
第三步:您必须添加新的普罗米修斯规则
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
creationTimestamp: null
labels:
prometheus: k8s
role: alert-rules
name: prometheus-podfail-rules
spec:
groups:
- name: ./podfail.rules
rules:
- alert: PodFailAlert
expr: sum(kube_pod_container_status_restarts_total{container="ffmpeggpu"}) BY (container) > 10
注意:角色应该是角色:在规则选择器prometheus类型中指定的alert-rules,要检查使用情况
kubectl get prometheus k8s -n monitoring -o yaml
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.