Kubernetes（K8s）-在一段时间内未创建新容器时，从Prometheus / Grafana获取警报

Question

I have a system in which a container is created every hour and I want to get an alert when a container is not created for a certain period of time from Prometheus or Grafana. Is there anyway to get a count of containers created in last let say X-hr and send an alert if the count of containers are less? The containers have a name template like : 2019-month-date-hour
Eg: 2019-5-11-23

I tried to use this query to get the count of containers created in a month count(count(container_last_seen{container_name=~"2019-5.*", namespace=$namespace}) by (container_name)) but not able to set grafana alert on this query for a period of time.

Answer 1

This is what you should do

1. use event exporter to capture the events generated in kubernetes cluster
2. get those events in promethues
3. create prometheus rule to check for container/pod creation event
4. get prometheus invoke alert via alert manager using the rule