metrics.num.samples kafka 设置如何影响 records-lag-max 的计算方式？

Question

When retrieving the consumer metrics via the kafka consumer API the records-lag-max is returning 0 occasionally even though describing the consumer group using the kafka-consumer-groups.sh tool shows that the consumer is lagging. The metrics.sample.window.ms is set to 30s by default and the metrics.num.samples set to 2 by default. How does the metrics.num.samples affect this calculation of records-lag-max?

Answer 1

i dont think it has any effect.

records-lag-* is driven by latest-offset field of the fetch response struct, which tells the client where the "edge" of a partition is on the broker. the client then compares that offset to the latest offset returned in the actual fetch response to figure out how many records it is "behind" the broker.

these metrics update every time a fetch response is received from broker(s), and if you query them before any such fetch responses have arrived you may see 0.

on the other hand kafka-consumer-groups shows you not where any particular consumer "is" (latest offsets delivered to that consumer), but what offsets that consumer last checkpointed to kafka. consumers checkpoint offsets periodically (typically once every 30-60 seconds) and so describing a consumer group can show non-0 lag where the metrics on the consumers themselves (which are updated more frequently) will show 0.