Prometheus to scrape K8s Ingress Endpoints

Question

My Prometheus instance is currently deployed outside to the K8s cluster. I went through the below article : https://www.linkedin.com/pulse/monitoring-kubernetes-prometheus-outside-cluster-steven-acreman

and this gives a very good idea about accessing the K8s services via the api server. I am still trying to make it work (stuck with certificates issue).

However, my question is can't we use Prometheus to scape the ingress-controller hostname to scrape metrics endpoint. Or is that even a standard approach?

I could not find a single document or implementation around the same. Please suggest if anyone has implemented the above scenario or has any docs on it.

Answer 1

I would use Kubernetes SD configurations , and take adventage of roles to discover targets.

Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API and always staying synchronized with the cluster state.

You could use role type endpoints :

The endpoints role discovers targets from listed endpoints of a service. For each endpoint address one target is discovered per port. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

Available meta labels:

• __meta_kubernetes_namespace : The namespace of the endpoints object.
• __meta_kubernetes_endpoints_name : The names of the endpoints object.
• For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
  • __meta_kubernetes_endpoint_hostname : Hostname of the endpoint.
  • __meta_kubernetes_endpoint_node_name : Name of the node hosting the endpoint.
  • __meta_kubernetes_endpoint_ready : Set to true or false for the endpoint's ready state.
  • __meta_kubernetes_endpoint_port_name : Name of the endpoint port.
  • __meta_kubernetes_endpoint_port_protocol : Protocol of the endpoint port.
  • __meta_kubernetes_endpoint_address_target_kind : Kind of the endpoint address target.
  • __meta_kubernetes_endpoint_address_target_name : Name of the endpoint address target.
• If the endpoints belong to a service, all labels of the role: service discovery are attached.
• For all targets backed by a pod, all labels of the role: pod discovery are attached.

Or role type ingress :

The ingress role discovers a target for each path of each ingress. This is generally useful for blackbox monitoring of an ingress. The address will be set to the host specified in the ingress spec.

Available meta labels:

• __meta_kubernetes_namespace : The namespace of the ingress object.
• __meta_kubernetes_ingress_name : The name of the ingress object.
• __meta_kubernetes_ingress_label_<labelname> : Each label from the ingress object.
• __meta_kubernetes_ingress_labelpresent_<labelname> : true for each label from the ingress object.
• __meta_kubernetes_ingress_annotation_<annotationname> : Each annotation from the ingress object.
• __meta_kubernetes_ingress_annotationpresent_<annotationname> : true for each annotation from the ingress object.
• __meta_kubernetes_ingress_scheme : Protocol scheme of ingress, https if TLS config is set. Defaults to http .
• __meta_kubernetes_ingress_path : Path from ingress spec. Defaults to / .

If you wish to check detailed example of configuring Prometheus for Kubernetes please see this file .

Also you might be interested in 3rd party Prometheus Operator , which automates the setup of Peometheus on top of Kubernetes.