Kafka set the maximum number of messages to read from the topic

Question

I am new to Apache Kafka and exploring the SimpleConsumer to read messages from the topic.

I use the following piece of code to do the same,

FetchRequestBuilder builder = new FetchRequestBuilder();
FetchRequest fetchRequest = builder.addFetch(topic, partitionId, offset, 1024).build();
FetchResponse fetchResponse;
try {
     fetchResponse = consumer.fetch(fetchRequest);
 } catch (Exception e) {}

This reads all the available messages in the specific partition; I would like to set the maximum number of messages to be read. Is there a way to do this at this stage? When there are larger number of messages in the queue, i don't want all of them landing in the JVM heap.

Another question,

The following code return a ByteBufferMessageSet.

fetchResponse.messageSet(topic, partitionId);

Does this mean, not all available messages actually land in memory?

Answer 1

While you can't limit the number of messages , you can limit the number of bytes received per topic-partition per request. However, this should be done as a configuration setting rather than as part of your consumer implementation code. The Kafka consumer config docs say that you can specify a maximum number of bytes read as socket.receive.buffer.bytes . This should allow you to have more fine-grained control over exactly how much space Kafka messages take up in the JVM heap. Please note that this value must be equal to or greater than the maximum message size on the broker, or else the producer could send messages that are too large to ever be consumed.

Answer 2

max.poll.records

The maximum number of records returned in a single call to poll(). https://kafka.apache.org/documentation/#consumerconfigs