使用度量 spring_cloud_stream_binder_kafka_offset 来计算 kafka 滞后

Question

I have an application consuming kafka messages for a long time without any restart. Also I have a dash that monitors the consumer lag by the property " spring_cloud_stream_binder_kafka_offset ".

When I had to restart it recently, I realized that some topics that I haven't sent messages for more than a month, started reporting some "strange values" on the same metric. I ran the following command to check the lags on the topic and then I realized this topic with "strange values" the column current-offset was empty "-", but i'm sure that this messages were processed in the past.

kafka-consumer-groups --bootstrap-server <server> --group group-test --describe --offsets --command-config 

GROUP	TOPIC	PARTITION	CURRENT-OFFSET	LOG-END-OFFSET	LAG
group-test	TEST	1	6468	6468	0
group-test	TEST	2	6396	6396	0
group-test	TEST1	0	-	88	-
group-test	TEST1	1	-	78	-

As I know, and I might be wrong, that this information about the current offset is stored inside a kafka topic. This topic also have an expire within 7 days since nothing is consumed.

My doubt is this: As the current offset has been "cleand up", shouldnt the property " spring_cloud_stream_binder_kafka_offset " reflects somehow this? Because its leading to some misunderstanding about the metric, since I don't have this lag.

Answer 1

See offset.retention.minutes . https://kafka.apache.org/documentation/#brokerconfigs_offsets.retention.minutes

It is 7 days by default, meaning the offsets are retained for 7 days after the last consumer left the group (with recent broker versions - since 2.1 if I recall correctly).

When the offsets are removed, it is as if the group never consumed anything from the topic/partition and, if started again, it will receive all the records in the topic/partition.