Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

Question

I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:

• 8080 for "business" http requests
• 8081/manage for actuator. So, I can access http://host:8081/manage/prometheus and see the metrics when running the process locally (without kubernetes).

Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).

I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.

But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:

FROM @docker.registry.address@/prom/prometheus:v2.15.2

COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh

ADD ./prometheus.yml  /etc/prometheus/

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh looks like:

#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                --storage.tsdb.path=/prometheus \
                --storage.tsdb.retention.time=3d \
                --web.console.libraries=/etc/prometheus/console_libraries \
                --web.console.templates=/etc/prometheus/consoles

My question is about how exactly I should define prometheus.yml so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).

I've started with ( prometheus.yml ):

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:

  - job_name: 'prometheus'
    metrics_path: /manage/prometheus
    kubernetes_sd_configs:
      - role: pod
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token  
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: sample-pod-app(.*)|another-pod-app(.*)

But apparently it doesn't work, so I've asking for the advices:

• If someone has a working example it would be the best :)
• Intuitively I understand I need to specify the port mapping for my 8081 port but I'm not exactly know how
• Since prometheus is supposed to run on another port am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
• Do I need any security related resources in kubernetes to be defined?

As a side note. At this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.

Answer 1

Rather than hardcoding it in prometheus config you need to make use of annotations on your pods to to tell prometheus which pods, what path and which port Prometheus should scrape.

prometheus.io/scrape: "true"
prometheus.io/path=/manage/prometheus
prometheus.io/port=8081
prometheus.io/scheme=http

Spring boot micrometer example with Prometheus on kubernetes. Prometheus deployment guide .

Answer 2

In order to let Prometheus collect metrics from your Spring Boot Application you need to add a specific dependencies to it. Here you can find a guide showing how to make it done: Spring Boot metrics monitoring using Prometheus & Grafana . Here is an example:

<dependency>  
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_spring_boot</artifactId>
            <version>0.1.0</version>
        </dependency>

        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_hotspot</artifactId>
            <version>0.1.0</version>
        </dependency>

If you would like to use a bit different strategy you can also check out this one: Monitoring Spring Boot applications with Prometheus and Grafana :

In order to compare the performance of different JDKs for reactive Spring Boot services, I made a setup in which a Spring Boot application is wrapped in a Docker container. This makes it easy to create different containers for different JDKs with the same Spring Boot application running in it. The Spring Boot application exposes metrics to Prometheus. Grafana can read these metrics and allows to make nice visualizations from it. This blog post describes a setup to get you up and running in minutes.

Please let me know if that helped.