Record Kubernetes container resource utilization data

Question

I'm doing a perf test for web server which is deployed on EKS cluster . I'm invoking the server using jmeter with different conditions (like varying thread count, payload size, etc..). So I want to record kubernetes perf data with the timestamp so that I can analyze these data with my jmeter output ( JTL ).

I have been digging through the internet to find a way to record kubernetes perf data. But I was unable to find a proper way to do that.

Can experts please provide me a standard way to do this??

Note: I have a multi-container pod also.

Answer 1

In line with @Jonas comment

This is the quickest way of installing Prometheus in you K8 cluster. Added Details in the answer as it was impossible to put the commands in a readable format in Comment.

1. Add bitnami helm repo.

helm repo add bitnami https://charts.bitnami.com/bitnami

2. Install helmchart for promethus

helm install my-release bitnami/kube-prometheus

Installation output would be:

C:\Users\ameena\Desktop\shine\Article\K8\promethus>helm install my-release bitnami/kube-prometheus
NAME: my-release
LAST DEPLOYED: Mon Apr 12 12:44:13 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Watch the Prometheus Operator Deployment status using the command:

    kubectl get deploy -w --namespace default -l app.kubernetes.io/name=kube-prometheus-operator,app.kubernetes.io/instance=my-release

Watch the Prometheus StatefulSet status using the command:

    kubectl get sts -w --namespace default -l app.kubernetes.io/name=kube-prometheus-prometheus,app.kubernetes.io/instance=my-release

Prometheus can be accessed via port "9090" on the following DNS name from within your cluster:

    my-release-kube-prometheus-prometheus.default.svc.cluster.local

To access Prometheus from outside the cluster execute the following commands:

    echo "Prometheus URL: http://127.0.0.1:9090/"
    kubectl port-forward --namespace default svc/my-release-kube-prometheus-prometheus 9090:9090

Watch the Alertmanager StatefulSet status using the command:

    kubectl get sts -w --namespace default -l app.kubernetes.io/name=kube-prometheus-alertmanager,app.kubernetes.io/instance=my-release

Alertmanager can be accessed via port "9093" on the following DNS name from within your cluster:

    my-release-kube-prometheus-alertmanager.default.svc.cluster.local

To access Alertmanager from outside the cluster execute the following commands:

    echo "Alertmanager URL: http://127.0.0.1:9093/"
    kubectl port-forward --namespace default svc/my-release-kube-prometheus-alertmanager 9093:9093

4. Follow the commands to forward the UI to localhost.

echo "Prometheus URL: http://127.0.0.1:9090/"
    kubectl port-forward --namespace default svc/my-release-kube-prometheus-prometheus 9090:9090

5. Open the UI in browser: http://127.0.0.1:9090/classic/graph

6. Annotate the pods for sending the metrics.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 4 # Update the replicas from 2 to 4
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9102'
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

7. In the ui put appropriate filters and start observing the crucial parameter such as memory CPU etc. UI supports autocomplete so it will not be that difficult to figure out things.

Regards