卡夫卡消费群未从单个分区读取

Question

I have a kafka topic with 3 partitions. I have a consumer group consuming from the said topic and I notice that 2 of 3 partitions are being consumed in real-time (lag 0), whereas 1 partition is simply not consumed at all.

How do I go about diagnosing this issue?

Appreciate inputs/comments.

Answer 1

To diagnose the issue you can use kafka-tool.

Following command shows partition, current offset, log-end-offset and lag client_id for particular group

./bin/kafka-consumer-groups.sh --bootstrap-server kafkaAddres:9092 --group yourGroupId --describe

Output will be something like that:

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
input           0          3               4               1               consumer-1-f64504ca-f514-4cef-95fa-1652702d4504 /192.168.160.1  consumer-1
input           1          3               3               0               consumer-1-f64504ca-f514-4cef-95fa-1652702d4504 /192.168.160.1  consumer-1
input           2          3               3               0               consumer-1-f64504ca-f514-4cef-95fa-1652702d4504 /192.168.160.1  consumer-1

Above output shows, that only one consumer is connected and read from all partitions. At time of checking for partitions 0 lag was 1.

NOTICE : Some of partitions might be empty

Answer 2

For your new partitions to reflect, you need to configure the metadata.max.age.ms property for your consumer.

The metadata contains the partitions for a given topic and which brokers are leader for those partitions etc. This is fetched by the consumer when the first poll occurs and is next fetched after the timeout or some error occurs.

By default the timeout is 300000ms = 5 minutes , so until 5 minutes your consumer cannot know the new partitions.

Metadata is refreshed either when there is an error or the metadata max age is reached.

consumerProperties.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG,"60000");

So try setting it to some thing low.