how to limit number of targets in prometheus

Question

I am wondering is there a way to limit the number of targets in prometheus. Searched prometheus documentation, but couldn't find such one. We are deploying prometheus using operators on k8s cluster and targets are added using servicemonitor crd. These yaml files are written by devs and I as a infra team can not control addition done by dev people. I am looking for global option in prometheus which can limit the addition of targets done in prometheus and throw error on prometheus dashboard saying something like upper limit has been reached.

Prometheus version-2.7

Answer 1

You cannot limit the number of targets; this would make a very odd feature. But you can monitor the number of targets scraped by Prometheus and trigger an error whenever this number exceeds a given threshold.

- alert: TooManyTargetsInPrometheus
  expr: count(up) > 42
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus instance has too much jobs"
    description: "Prometheus has {{ $value }} targets which is way too much."

If you want automation, you can trigger an action on the alert to rollback the configuration or downsize the offending processes.

In my opinion, you would alert if your rate of missed scrape start to increase or on the number of metrics ingested by seconds exceeds recommendations, not on a arbitrary limit.

Answer 2

You are looking for a ServiceMonitor property called 'targetLimit'.

Here is an example of how to use it:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: service-scraper
  labels:
    release: kube-prometheus-stack-1606315657
spec:
  selector:
    matchLabels:
      monitor: 'true'
  targetLimit: 1
  endpoints:
  - port: web
    path: /metrics

Go to Prometheus dashboard -> Status -> Targets. You will find this error if there are more replicas than targetLimit allows.

Check serviceMonitor spec

Answer 3

As of Prometheus v2.21 there is a configuration option target_limit under scrape_config .

(noted as experimental, may change in future)