Service Level metrics Prometheus in k8

Question

would like to see k8 Service level metrics in Grafana from underlying prometheus server.

For instance:

1) If i have 3 application pods exposed through a service i would like to see service level metrics for CPU,memory & network I/O pressure ,Total # of requests,# of requests failed

2)Also if i have group of pods(replicas) related to an application which doesn"t have Service on top of them would like to see the aggregated metrics of the pods related to that application in a single view on grafana

What would be the prometheus queries to achieve the same

Answer 1

Service level metrics for CPU, memory & network I/O pressure

If you have Prometheus installed on your Kubernetes cluster, all those statistics are being already collected by Prometheus. There are many good articles about how to install and how to use Kubernetes+Prometheus, try to check that one, as an example.

Here is an example of a request to fetch container memory usage:

container_memory_usage_bytes{image="CONTAINER:VERSION"}

Total # of requests,# of requests failed

Those are service-level metrics, and for collecting them, you need to use Prometheus Exporter created especially for your service. Check the list with exporters, find one which you need for your service and follow its instruction.

If you cannot find an Exporter for your application, you can write it yourself, here is an official documentation about it.

application which doesn"t have Service on top of them would like to see the aggregated metrics of the pods related to that application in a single view on grafana

It is possible to combine any graphics in a single view in Grafana using Dashboards and Panels. Check an official documentation , all that topics pretty detailed and easy to understand.

Aggregation can be done by Prometheus itself by aggregation operations .

All metrics from Kubernetes has labels , so you can group by them:

sum(http_requests_total) by (application, group) , where application and group is labels.

Also, here is an official Prometheus instruction about how to add Prometheus to Grafana as a Datasourse .