Kafka not sending enough messages to consumer

Question

I've a kafka topic, 3 partitions, only one consumer with batch. I am using spring kafka on the consumer side with following consumer props:

max.poll.records=10000
fetch.min.bytes=2000000
fetch.max.bytes=15000000
fetch.max.wait.ms=1000
max.poll.interval.ms=300000
auto.offset.reset.config=earliest
idle.event.interval=120000

Even tho there are thousands of messages (GBs of data) waiting in the queue, kafka consumer receives around 10 messages (total size around 1MB) on each poll. The consumer should fetch batches of fetch.max.bytes (in my prop ~15MB) or max.poll.records (10000 in my case) . What's the problem?

Answer 1

There are several scenarios which may cause this, do the following changes:

1. Increase fetch.min.bytes - the consumer also may fetch batches of fetch.min.bytes , which is 1.9MB.
2. Increase fetch.max.wait.ms - the poll function waits for fetch.min.bytes or fetch.max.wait.ms to trigger, whatever comes first.
   fetch.max.wait.ms is 1 second in your configuration, sounds alright but increase it just in case this is the problem.
3. Increase max.partition.fetch.bytes - default is 1MB, it can decrease the poll size for small partitioned topics like yours (limit up to 3MB poll for 3 partitions topic with a single consumer).

Try to use these values:

fetch.min.bytes=12000000
fetch.max.wait.ms=5000  
max.partition.fetch.bytes=5000000  

Deeper explanation:
https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html