Prometheus 是否可以在大型批处理作业中捕获每个进程的指标？

Question

As per title, is it possible for Prometheus to capture metrics of each individual process in a large batch job?

This job runs hourly and processes records at a rate of about 500-1000 records/second, sending gauge metrics (unique to each record) to statsd_exporter for monitoring our SLOs.

however, i realized Prometheus can only capture as much as the scrape_interval allows, meaning its definitely missing some values sent to statsd_exporter (eg possible missing some spikes in value).

Is there a way to overcome this? or perhaps i should be looking at some other tools instead.

updates: provided example of metrics sent. job label is limited to 10 different labels ie 10 timeseries

# HELP time_taken_gauge Time taken for a particular job type to finish processing a record.
# TYPE time_taken_gauge gauge
time_taken_gauge{job="a"} 123
time_taken_gauge{job="b"} 1314
time_taken_gauge{job="c"} 5435
time_taken_gauge{job="d"} 212
time_taken_gauge{job="e"} 231
time_taken_gauge{job="f"} 324
time_taken_gauge{job="g"} 15
time_taken_gauge{job="h"} 1213
time_taken_gauge{job="i"} 123
time_taken_gauge{job="j"} 1235

Only challenge is these are sent in at a much higher rate than Prometheus' scrape interval (1s), hence missing some records

time_taken_gauge{job="a"} 123
time_taken_gauge{job="a"} 1232 <- scraped
time_taken_gauge{job="a"} 12412
time_taken_gauge{job="a"} 53453 <- high value metric missed but potentially problematic
time_taken_gauge{job="a"} 1564
time_taken_gauge{job="a"} 756
time_taken_gauge{job="a"} 34 <- scraped
time_taken_gauge{job="a"} 15433
.
.
.
time_taken_gauge{job="a"} 345 <- scraped

Answer 1

500-1000 records/second, sending gauge metrics (unique to each record)

That's going to be high cardinality. If you want a unique metric per record then you need an event logging system like ELK, not metrics systems like Prometheus or whatever you have statsd feeding into.

Answer 2

The way to overcome short-lived jobs, or slow jobs is to use a Pushgateway, which will store all metrics pushes to it and expose them to be scraped.

This is the standard for handling such cases, I'm not familiar with alternatives..Be happy to.

The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.

https://github.com/prometheus/pushgateway