Apache Kafka- Understanding the relationship between timeout present as a parameter in the poll() method and fetch.max.wait.ms

Question

I wanted to understand the relationship between the timeout present in the poll() method and the configuration fetch.max.wait.ms. So , lets say I have the following configuration

          fetch.min.bytes= 50
          fetch.max.wait.ms= 400 ms
          timeout on poll method= 200 ms

So, consider I call the poll method with the above specified timeout. The Consumer sends a fetch request to the Kafka Broker who is the leader for that partition. The Broker has not got enough bytes to send according to the configuration fetch.min.bytes, so it will wait for maximum of 400 milliseconds to respond for enough data to arrive. But I have configured, the timeout to be 200 ms for the poll method, so does that mean, behind the hood, when the fetch request is sent, it only waits for 200 ms for the server to respond and then terminates the connection?

Is that how it will turn out? In this scenario, would it be safe to say, you would always configure your timeout to be -

         timeout >= network latency + fetch.max.wait.ms

Also, does Kafka Consumer fetch records proactively? By that I mean, is the consumer busy fetching records behind the hood , when the user code is busy processing the records on the last poll() method call, so that to reduce latency when the next time poll() is called. If yes, how does it maintain this internal buffer? Can we also configure, the maximum size of this internal buffer?

Thank you in advance.

Answer 1

Time out on poll allows you to do asynchronous processing. After subscribing to a set of topics, the consumer will automatically join the group when poll(long) is invoked. The poll API is designed to ensure consumer availability.

As long as the consumer continue to call poll, the consumer will stay in the group and continue to receive messages from the partitions it was assigned.

Under the hood, the consumer sends periodic heartbeats to the server. If the consumer crashes or is unable to send heartbeats for a duration of session.timeout.ms , then the consumer will be considered dead and its partitions will be reassigned.

But we should be careful that the long value in the poll(long) is not too long. This makes the whole process synchronous. You can read the discussion in the below link.

https://github.com/confluentinc/confluent-kafka-dotnet/issues/142

fetch.max.wait.ms This will make sure whenever a fetch request is created the server will block the request until the time specified. This usually kicks in if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes .

Point 1 : When there is a fetch request the server blocks your fetch request for 400ms if it does not meet 50bytes.

fetch.min.bytes= 50
fetch.max.wait.ms= 400 ms

Point 2 : For every 200ms you consumer sends a heartbeat to avoid rebalance by kafka.

timeout on poll method= 200 ms

When Point 1 happens your consumer is idle but since you did Point 2 the heart beat is sent at every 200ms so rebalance does not occur and you may perform some tasks asynchronously for the next 200ms.

So setting poll() will only make sure that your consumer is not considered dead and fetch.max.wait.ms is to tell the server about how long it need to wait when the fetch request comes. What i mean to say is there is not inherent dependency on the two parameter. poll() is more of the asynchronous way of doing things in your code.

Timeout is based purely on the poll().