使用 Prometheus 的平均请求持续时间

Question

I have deployed the Istio Bookinfo application in a Kubernetes cluster. Following the documentation, I'm trying to measure the average request duration using the following query:

rate(istio_request_duration_milliseconds_sum[1m]) / rate(istio_request_duration_milliseconds_count[1m])

This query returns me "doubled" results ie for each request I get two results, with different Value but same source and destination , one labelled as reporter="source" and the other as reporter="destination" . I was not able to find any clarification about it and I have not clear how these measures work. Why I have two values returned?

Answer 1

The metrics are indeed reported from two sources, once from the request sender and another time from the request receiver. This is a bit redundant, but in some situations the values differ and especially for requests duration: from request sender ( reporter=source ), the duration stands for the whole request latency (server processing time + network roundtrip) whereas from the request receiver ( reporter=destination ), it stands mainly for server processing time.

In a few special situations there is no redundancy and metrics are only reported once (be it from source or from destination): eg when istio features such as fault injection or traffic mirroring come into play.

As a result, in Prometheus you should always filter all your queries with either {reporter="source"} or {reporter="destination"} .