How to graph individual Summary metric instances in Prometheus?

Question

I'm using Prometheus' Summary metric to collect the latency of an API call. Instead of making an actual API call, I'm simply calling Thread.sleep(1000) to simulate a 1 second api-call latency value -- this makes the Summary hold a value of .01 (for 1 second of latency). But if, for example, I invoke Thread.sleep(1000) twice in the same minute, the Summary metric ends up with a value of .02 (for 2 seconds of latency), instead of two individual instances of .01 latency that just happened to occur within the same minute. My problem is the Prometheus query. The Prometheus query I am currently using is: rate(my_custom_summary_sum[1m]) .

What should my Prometheus query be, such that I can see the latency of each individual Thread.sleep(1000) invocation . As of right now, the Summary metric collects and displays the total latency sum per minute. How can I display the latency of each individual call to Thread.sleep(1000) (ie the API request)?

private static final Summary mySummary = Summary.build()
                                        .name("my_custom_summary")
                                        .help("This is a custom summary that keeps track of latency")
                                        .register();

Summary.Timer requestTimer = mySummary.startTimer(); //starting timer for mySummary 'Summary' metric
        Thread.sleep(1000); //sleep for one second
        requestTimer.observeDuration(); //record the time elapsed

This is the graph that results from this query: Prometheus graph

Answer 1

Prometheus is a metrics-based monitoring system, it cares about overall performance and behaviour - not individual requests.

What you are looking for is a logs-based system, such as Graylog or the ELK stack.