Prometheus alertmanager fails to send notifications due to “context deadline exceeded”"

Question

I configured prometheus-operator chart with prometheus-msteams for monitoring and alerting of k8s cluster.

But all notifications are not correctly directed to the MSteams channel. If i have 6 alerts that are firing, i can see them in the alertmanager's UI, but only one or two of them are sent to MS teams channel.

I can see this log in alertmanager pod:

C:\monitoring>kubectl logs alertmanager-monitor-prometheus-operato-alertmanager-0 -c alertmanager

level=info ts=2019-11-04T09:16:47.358Z caller=main.go:217 msg="Starting Alertmanager" version="(version=0.19.0, branch=HEAD, revision=7aa5d19fea3f58e3d27dbdeb0f2883037168914a)"
level=info ts=2019-11-04T09:16:47.358Z caller=main.go:218 build_context="(go=go1.12.8, user=root@587d0268f963, date=20190903-15:01:40)"
level=warn ts=2019-11-04T09:16:47.553Z caller=cluster.go:228 component=cluster msg="failed to join cluster" err="1 error occurred:\n\t* Failed to resolve alertmanager-monitor-prometheus-operato-alertmanager-0.alertmanager-operated.monitoring.svc:9094: lookup alertmanager-monitor-prometheus-operato-alertmanager-0.alertmanager-operated.monitoring.svc on 169.254.25.10:53: no such host\n\n"
level=info ts=2019-11-04T09:16:47.553Z caller=cluster.go:230 component=cluster msg="will retry joining cluster every 10s"
level=warn ts=2019-11-04T09:16:47.553Z caller=main.go:308 msg="unable to join gossip mesh" err="1 error occurred:\n\t* Failed to resolve alertmanager-monitor-prometheus-operato-alertmanager-0.alertmanager-operated.monitoring.svc:9094: lookup alertmanager-monitor-prometheus-operato-alertmanager-0.alertmanager-operated.monitoring.svc on 169.254.25.10:53: no such host\n\n"
level=info ts=2019-11-04T09:16:47.553Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2019-11-04T09:16:47.597Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2019-11-04T09:16:47.598Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2019-11-04T09:16:47.601Z caller=main.go:466 msg=Listening address=:9093
level=info ts=2019-11-04T09:16:49.554Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000149822s
level=info ts=2019-11-04T09:16:57.555Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001110685s

level=error ts=2019-11-04T09:38:02.472Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager" context_err="context deadline exceeded"
level=error ts=2019-11-04T09:38:02.472Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager"
level=error ts=2019-11-04T09:43:02.472Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager" context_err="context deadline exceeded"
level=error ts=2019-11-04T09:43:02.472Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager"
level=error ts=2019-11-04T09:48:02.473Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager" context_err="context deadline exceeded"
level=error ts=2019-11-04T09:48:02.473Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager"
level=error ts=2019-11-04T09:53:02.473Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager" context_err="context deadline exceeded"
level=error ts=2019-11-04T09:53:02.473Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="unexpected status code 500: http://prometheus-msteams:2000/alertmanager"

How can I solve this error?

EDIT :

The setup uses prometheus-msteams as a webhook to redirect the alerts notifications from alertmanager to MSTeams channel.

The prometheus-msteams container logs also have some errors:

C:\> kubectl logs prometheus-msteams-564bc7d99c-dpzsm

time="2019-11-06T06:45:14Z" level=info msg="Version: v1.1.4, Commit: d47a7ab, Branch: HEAD, Build Date: 2019-08-04T17:17:06+0000"
time="2019-11-06T06:45:14Z" level=info msg="Parsing the message card template file: /etc/template/card.tmpl"
time="2019-11-06T06:45:15Z" level=warning msg="If the 'config' flag is used, the 'webhook-url' and 'request-uri' flags will be ignored."
time="2019-11-06T06:45:15Z" level=info msg="Parsing the configuration file: /etc/config/connectors.yaml"
time="2019-11-06T06:45:15Z" level=info msg="Creating the server request path \"/alertmanager\" with webhook \"https://outlook.office.com/webhook/00ce0266-7013-4d53-a20f-115ece04042d@9afb1f8a-2192-45ba-b0a1-6b193c758e24/IncomingWebhook/43c3d745ff5e426282f1bc6b5e79bfea/8368b12d-8ac9-4832-b7b5-b337ac267220\""
time="2019-11-06T06:45:15Z" level=info msg="prometheus-msteams server started listening at 0.0.0.0:2000"

time="2019-11-06T07:01:07Z" level=info msg="/alertmanager received a request"
time="2019-11-06T07:01:07Z" level=debug msg="Prometheus Alert: {\"receiver\":\"prometheus-msteams\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"KubeDeploymentReplicasMismatch\",\"deployment\":\"storagesvc\",\"endpoint\":\"http\",\"instance\":\"10.233.108.72:8080\",\"job\":\"kube-state-metrics\",\"namespace\":\"fission\",\"pod\":\"monitor-kube-state-metrics-856bc9455b-7z5qx\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"service\":\"monitor-kube-state-metrics\",\"severity\":\"critical\"},\"annotations\":{\"message\":\"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"},\"startsAt\":\"2019-11-06T07:00:32.453590324Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://monitor-prometheus-operato-prometheus.monitoring:9090/graph?g0.expr=kube_deployment_spec_replicas%7Bjob%3D%22kube-state-metrics%22%7D+%21%3D+kube_deployment_status_replicas_available%7Bjob%3D%22kube-state-metrics%22%7D\\u0026g0.tab=1\"},{\"status\":\"firing\",\"labels\":{\"alertname\":\"KubePodNotReady\",\"namespace\":\"fission\",\"pod\":\"storagesvc-5bff46b69b-vfdrd\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"severity\":\"critical\"},\"annotations\":{\"message\":\"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"},\"startsAt\":\"2019-11-06T07:00:32.453590324Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://monitor-prometheus-operato-prometheus.monitoring:9090/graph?g0.expr=sum+by%28namespace%2C+pod%29+%28kube_pod_status_phase%7Bjob%3D%22kube-state-metrics%22%2Cphase%3D~%22Failed%7CPending%7CUnknown%22%7D%29+%3E+0\\u0026g0.tab=1\"}],\"groupLabels\":{\"namespace\":\"fission\",\"severity\":\"critical\"},\"commonLabels\":{\"namespace\":\"fission\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"severity\":\"critical\"},\"commonAnnotations\":{},\"externalURL\":\"http://monitor-prometheus-operato-alertmanager.monitoring:9093\",\"version\":\"4\",\"groupKey\":\"{}:{namespace=\\\"fission\\\", severity=\\\"critical\\\"}\"}"
time="2019-11-06T07:01:07Z" level=debug msg="Alert rendered in template file: \r\n{\r\n  \"@type\": \"MessageCard\",\r\n  \"@context\": \"http://schema.org/extensions\",\r\n  \"themeColor\": \"8C1A1A\",\r\n  \"summary\": \"\",\r\n  \"title\": \"Prometheus Alert (firing)\",\r\n  \"sections\": [ \r\n    {\r\n      \"activityTitle\": \"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\r\n      \"facts\": [\r\n        {\r\n          \"name\": \"message\",\r\n          \"value\": \"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\"\r\n        },\r\n        {\r\n          \"name\": \"runbook\\\\_url\",\r\n          \"value\": \"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"\r\n        },\r\n        {\r\n          \"name\": \"alertname\",\r\n          \"value\": \"KubeDeploymentReplicasMismatch\"\r\n        },\r\n        {\r\n          \"name\": \"deployment\",\r\n          \"value\": \"storagesvc\"\r\n        },\r\n        {\r\n          \"name\": \"endpoint\",\r\n          \"value\": \"http\"\r\n        },\r\n        {\r\n          \"name\": \"instance\",\r\n          \"value\": \"10.233.108.72:8080\"\r\n        },\r\n        {\r\n          \"name\": \"job\",\r\n          \"value\": \"kube-state-metrics\"\r\n        },\r\n        {\r\n          \"name\": \"namespace\",\r\n          \"value\": \"fission\"\r\n        },\r\n        {\r\n          \"name\": \"pod\",\r\n          \"value\": \"monitor-kube-state-metrics-856bc9455b-7z5qx\"\r\n        },\r\n        {\r\n          \"name\": \"prometheus\",\r\n          \"value\": \"monitoring/monitor-prometheus-operato-prometheus\"\r\n        },\r\n        {\r\n          \"name\": \"service\",\r\n          \"value\": \"monitor-kube-state-metrics\"\r\n        },\r\n        {\r\n          \"name\": \"severity\",\r\n          \"value\": \"critical\"\r\n        }\r\n      ],\r\n      \"markdown\": true\r\n    },\r\n    {\r\n      \"activityTitle\": \"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\r\n      \"facts\": [\r\n        {\r\n          \"name\": \"message\",\r\n          \"value\": \"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\"\r\n        },\r\n        {\r\n          \"name\": \"runbook\\\\_url\",\r\n          \"value\": \"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"\r\n        },\r\n        {\r\n          \"name\": \"alertname\",\r\n          \"value\": \"KubePodNotReady\"\r\n        },\r\n        {\r\n          \"name\": \"namespace\",\r\n          \"value\": \"fission\"\r\n        },\r\n        {\r\n          \"name\": \"pod\",\r\n          \"value\": \"storagesvc-5bff46b69b-vfdrd\"\r\n        },\r\n        {\r\n          \"name\": \"prometheus\",\r\n          \"value\": \"monitoring/monitor-prometheus-operato-prometheus\"\r\n        },\r\n        {\r\n          \"name\": \"severity\",\r\n          \"value\": \"critical\"\r\n        }\r\n      ],\r\n      \"markdown\": true\r\n    }\r\n  ]\r\n}\r\n"
time="2019-11-06T07:01:07Z" level=debug msg="Size of message is 1714 Bytes (~1 KB)"
time="2019-11-06T07:01:07Z" level=info msg="Created a card for Microsoft Teams /alertmanager"
time="2019-11-06T07:01:07Z" level=debug msg="Teams message cards: [{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"8C1A1A\",\"summary\":\"\",\"title\":\"Prometheus Alert (firing)\",\"sections\":[{\"activityTitle\":\"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\"facts\":[{\"name\":\"message\",\"value\":\"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"},{\"name\":\"alertname\",\"value\":\"KubeDeploymentReplicasMismatch\"},{\"name\":\"deployment\",\"value\":\"storagesvc\"},{\"name\":\"endpoint\",\"value\":\"http\"},{\"name\":\"instance\",\"value\":\"10.233.108.72:8080\"},{\"name\":\"job\",\"value\":\"kube-state-metrics\"},{\"name\":\"namespace\",\"value\":\"fission\"},{\"name\":\"pod\",\"value\":\"monitor-kube-state-metrics-856bc9455b-7z5qx\"},{\"name\":\"prometheus\",\"value\":\"monitoring/monitor-prometheus-operato-prometheus\"},{\"name\":\"service\",\"value\":\"monitor-kube-state-metrics\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true},{\"activityTitle\":\"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\"facts\":[{\"name\":\"message\",\"value\":\"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"},{\"name\":\"alertname\",\"value\":\"KubePodNotReady\"},{\"name\":\"namespace\",\"value\":\"fission\"},{\"name\":\"pod\",\"value\":\"storagesvc-5bff46b69b-vfdrd\"},{\"name\":\"prometheus\",\"value\":\"monitoring/monitor-prometheus-operato-prometheus\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true}]}]"
time="2019-11-06T07:01:07Z" level=info msg="Microsoft Teams response text: 1"
time="2019-11-06T07:01:07Z" level=info msg="A card was successfully sent to Microsoft Teams Channel. Got http status: 200 OK"
time="2019-11-06T07:01:07Z" level=info msg="Microsoft Teams response text: Summary or Text is required."
time="2019-11-06T07:01:07Z" level=error msg="Failed sending to the Teams Channel. Teams http response: 400 Bad Request"

time="2019-11-06T07:01:08Z" level=info msg="/alertmanager received a request"
time="2019-11-06T07:01:08Z" level=debug msg="Prometheus Alert: {\"receiver\":\"prometheus-msteams\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"KubeDeploymentReplicasMismatch\",\"deployment\":\"storagesvc\",\"endpoint\":\"http\",\"instance\":\"10.233.108.72:8080\",\"job\":\"kube-state-metrics\",\"namespace\":\"fission\",\"pod\":\"monitor-kube-state-metrics-856bc9455b-7z5qx\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"service\":\"monitor-kube-state-metrics\",\"severity\":\"critical\"},\"annotations\":{\"message\":\"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"},\"startsAt\":\"2019-11-06T07:00:32.453590324Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://monitor-prometheus-operato-prometheus.monitoring:9090/graph?g0.expr=kube_deployment_spec_replicas%7Bjob%3D%22kube-state-metrics%22%7D+%21%3D+kube_deployment_status_replicas_available%7Bjob%3D%22kube-state-metrics%22%7D\\u0026g0.tab=1\"},{\"status\":\"firing\",\"labels\":{\"alertname\":\"KubePodNotReady\",\"namespace\":\"fission\",\"pod\":\"storagesvc-5bff46b69b-vfdrd\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"severity\":\"critical\"},\"annotations\":{\"message\":\"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"},\"startsAt\":\"2019-11-06T07:00:32.453590324Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://monitor-prometheus-operato-prometheus.monitoring:9090/graph?g0.expr=sum+by%28namespace%2C+pod%29+%28kube_pod_status_phase%7Bjob%3D%22kube-state-metrics%22%2Cphase%3D~%22Failed%7CPending%7CUnknown%22%7D%29+%3E+0\\u0026g0.tab=1\"}],\"groupLabels\":{\"namespace\":\"fission\",\"severity\":\"critical\"},\"commonLabels\":{\"namespace\":\"fission\",\"prometheus\":\"monitoring/monitor-prometheus-operato-prometheus\",\"severity\":\"critical\"},\"commonAnnotations\":{},\"externalURL\":\"http://monitor-prometheus-operato-alertmanager.monitoring:9093\",\"version\":\"4\",\"groupKey\":\"{}:{namespace=\\\"fission\\\", severity=\\\"critical\\\"}\"}"
time="2019-11-06T07:01:08Z" level=debug msg="Alert rendered in template file: \r\n{\r\n  \"@type\": \"MessageCard\",\r\n  \"@context\": \"http://schema.org/extensions\",\r\n  \"themeColor\": \"8C1A1A\",\r\n  \"summary\": \"\",\r\n  \"title\": \"Prometheus Alert (firing)\",\r\n  \"sections\": [ \r\n    {\r\n      \"activityTitle\": \"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\r\n      \"facts\": [\r\n        {\r\n          \"name\": \"message\",\r\n          \"value\": \"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\"\r\n        },\r\n        {\r\n          \"name\": \"runbook\\\\_url\",\r\n          \"value\": \"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"\r\n        },\r\n        {\r\n          \"name\": \"alertname\",\r\n          \"value\": \"KubeDeploymentReplicasMismatch\"\r\n        },\r\n        {\r\n          \"name\": \"deployment\",\r\n          \"value\": \"storagesvc\"\r\n        },\r\n        {\r\n          \"name\": \"endpoint\",\r\n          \"value\": \"http\"\r\n        },\r\n        {\r\n          \"name\": \"instance\",\r\n          \"value\": \"10.233.108.72:8080\"\r\n        },\r\n        {\r\n          \"name\": \"job\",\r\n          \"value\": \"kube-state-metrics\"\r\n        },\r\n        {\r\n          \"name\": \"namespace\",\r\n          \"value\": \"fission\"\r\n        },\r\n        {\r\n          \"name\": \"pod\",\r\n          \"value\": \"monitor-kube-state-metrics-856bc9455b-7z5qx\"\r\n        },\r\n        {\r\n          \"name\": \"prometheus\",\r\n          \"value\": \"monitoring/monitor-prometheus-operato-prometheus\"\r\n        },\r\n        {\r\n          \"name\": \"service\",\r\n          \"value\": \"monitor-kube-state-metrics\"\r\n        },\r\n        {\r\n          \"name\": \"severity\",\r\n          \"value\": \"critical\"\r\n        }\r\n      ],\r\n      \"markdown\": true\r\n    },\r\n    {\r\n      \"activityTitle\": \"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\r\n      \"facts\": [\r\n        {\r\n          \"name\": \"message\",\r\n          \"value\": \"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\"\r\n        },\r\n        {\r\n          \"name\": \"runbook\\\\_url\",\r\n          \"value\": \"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"\r\n        },\r\n        {\r\n          \"name\": \"alertname\",\r\n          \"value\": \"KubePodNotReady\"\r\n        },\r\n        {\r\n          \"name\": \"namespace\",\r\n          \"value\": \"fission\"\r\n        },\r\n        {\r\n          \"name\": \"pod\",\r\n          \"value\": \"storagesvc-5bff46b69b-vfdrd\"\r\n        },\r\n        {\r\n          \"name\": \"prometheus\",\r\n          \"value\": \"monitoring/monitor-prometheus-operato-prometheus\"\r\n        },\r\n        {\r\n          \"name\": \"severity\",\r\n          \"value\": \"critical\"\r\n        }\r\n      ],\r\n      \"markdown\": true\r\n    }\r\n  ]\r\n}\r\n"
time="2019-11-06T07:01:08Z" level=debug msg="Size of message is 1714 Bytes (~1 KB)"
time="2019-11-06T07:01:08Z" level=info msg="Created a card for Microsoft Teams /alertmanager"
time="2019-11-06T07:01:08Z" level=debug msg="Teams message cards: [{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"8C1A1A\",\"summary\":\"\",\"title\":\"Prometheus Alert (firing)\",\"sections\":[{\"activityTitle\":\"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\"facts\":[{\"name\":\"message\",\"value\":\"Deployment fission/storagesvc has not matched the expected number of replicas for longer than 15 minutes.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch\"},{\"name\":\"alertname\",\"value\":\"KubeDeploymentReplicasMismatch\"},{\"name\":\"deployment\",\"value\":\"storagesvc\"},{\"name\":\"endpoint\",\"value\":\"http\"},{\"name\":\"instance\",\"value\":\"10.233.108.72:8080\"},{\"name\":\"job\",\"value\":\"kube-state-metrics\"},{\"name\":\"namespace\",\"value\":\"fission\"},{\"name\":\"pod\",\"value\":\"monitor-kube-state-metrics-856bc9455b-7z5qx\"},{\"name\":\"prometheus\",\"value\":\"monitoring/monitor-prometheus-operato-prometheus\"},{\"name\":\"service\",\"value\":\"monitor-kube-state-metrics\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true},{\"activityTitle\":\"[](http://monitor-prometheus-operato-alertmanager.monitoring:9093)\",\"facts\":[{\"name\":\"message\",\"value\":\"Pod fission/storagesvc-5bff46b69b-vfdrd has been in a non-ready state for longer than 15 minutes.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready\"},{\"name\":\"alertname\",\"value\":\"KubePodNotReady\"},{\"name\":\"namespace\",\"value\":\"fission\"},{\"name\":\"pod\",\"value\":\"storagesvc-5bff46b69b-vfdrd\"},{\"name\":\"prometheus\",\"value\":\"monitoring/monitor-prometheus-operato-prometheus\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true}]}]"
time="2019-11-06T07:01:08Z" level=info msg="Microsoft Teams response text: Summary or Text is required."
time="2019-11-06T07:01:08Z" level=error msg="Failed sending to the Teams Channel. Teams http response: 400 Bad Request"

Maybe due to this 400 bad request error from prometheus-msteams, the alertmanager was returning unexpected status code 500 .

Answer 1

An issue with the file https://github.com/bzon/prometheus-msteams/blob/master/chart/prometheus-msteams/card.tmpl caused these errors.

The problem was the summary field was empty. A slight change in the file was made as described in this tutorial solved the errors.

You can use the new modified card template by overriding the default one.