实时监控k8s容器中的CPU使用率/ CPU限制

Question

I am trying to measure CPU usage of a container in Kubernetes, represented as a ratio between actual usage and usage limit in a short time window. This should be ideally close to real-time (up to 5s delay).

I have full control of the container code and I can also extend the pod with a sidecar container to do reporting for me.

I have looked at Prometheus deployed using Prometheus operator, but I am seeing the data landing with large delays or even not showing up at all for some pods.

I was hoping somebody could shed some light on how to implement any of those:

1. sidecar container that can query cpu usage/cpu limit and send the data to another service (I am worried that this is impossible, because containers run in isolated file systems).
2. another process within main container, that can do the reporting. Maybe dividing $(cat /sys/fs/cgroup/cpu/cpuacct.usage) / $(/sys/fs/cgroup/cpu/cpu.cfs_quota_us) would do the trick?
3. use some existing software tool/service to achieve this. Any recommendation would be appreciated.

Thank you very much!

Answer 1

Deploy a sidecar container along with the container that you want to monitor. The sidecar container should monitor the cpu of the main container and push the status to prometheus or some other monitoring service. With alerting you can set thresholds and if the cpu is over the threshold then prometheus would trigger an alert action through alert manager service