用于检查 AlertManager 的错误 Prometheus 端点

Question

I installed Prometheus (follow in this link: https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/ )

But, when checking status of Targets, it shows "Down" for AlertManager service, every another endpoint are up, please see the attached file

Then, I check Service Discovery, the discovered labels shows:

" address ="192.168.180.254:9093" __meta_kubernetes_endpoint_address_target_kind="Pod" __meta_kubernetes_endpoint_address_target_name="alertmanager-6c666985cc-54rjm" __meta_kubernetes_endpoint_node_name="worker-node1" __meta_kubernetes_endpoint_port_protocol="TCP" __meta_kubernetes_endpoint_ready="true" __meta_kubernetes_endpoints_name="alertmanager" __meta_kubernetes_namespace="monitoring" __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP="192.168.180.254/32" __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP="true" __meta_kubernetes_pod_container_name="alertmanager" __meta_kubernetes_pod_container_port_name="alertmanager" __meta_kubernetes_pod_container_port_number="9093""

But Target Labels show another port ( 8080 ), I don't know why:

 instance="192.168.180.254:8080" job="kubernetes-service-endpoints" kubernetes_name="alertmanager" kubernetes_namespace="monitoring"

Answer 1

First, if you want to install prometheus and grafana without getting sick, you need to do it though helm.

First install helm

And then

helm install installationWhatEverName stable/prometheus-operator

Answer 2

I've reproduced your issue on GCE.

If you are using version 1.16+ you have probably changed apiVersion as in tutorial you have Deployment in extensions/v1beta1 . Since K8s 1.16+ you need to change it to apiVersion: apps/v1 . Otherwise you will get error like:

error: unable to recognize "STDIN": no matches for kind "Deployment" in version "extensions/v1beta1"

Second thing, in 1.16+ you need to specify selector . If you will not do it you will receive another error:

`error: error validating "STDIN": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false`

It would look like:

...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
...

Regarding port 8080 please check this article with example.

Port: Port is the port number which makes a service visible to other services running within the same K8s cluster. In other words, in case a service wants to invoke another service running within the same Kubernetes cluster, it will be able to do so using port specified against “port” in the service spec file.

It worked for my environment in GCE. Did you configure firewall for your endpoints?

In addition. In Helm 3 some hooks were deprecated. You can find this information here .

If you still have issue please provide your YAMLs witch applied changes to version 1.16+.