How to get prometheus to monitor kubernetes service?

Question

I'd like to monitor my Kubernetes Service objects to ensure that they have > 0 Pods behind them in "Running" state.

However, to do this I would have to first group the Pods by service and then group them further by status.

I would also like to do this programatically (eg for each service in namespace ...)

There's already some code that does this in the Sensu kubernetes plugin: https://github.com/sensu-plugins/sensu-plugins-kubernetes/blob/master/bin/check-kube-service-available.rb but I haven't seen anything that shows how to do it with Prometheus.

Has anyone setup kubernetes service level health checks with Prometheus? If so, how did you group by service and then group by Pod status?

Answer 1

The examples I have seen for Prometheus service checks relied on the blackbox exporter:

The blackbox exporter will try a given URL on the service. If that succeeds, at least one pod is up and running.

See here for an example: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml in job kubernetes-service-endpoints

The URL to probe might be your liveness probe or something else. If your services don't talk HTTP, you can make the blackbox exporter test other protocols as well.