Spring Kafka configuration for high consumer processing time

Question

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.8.2</version>
</dependency>

I have a kafka consumer that takes anywhere from 30-120 seconds to process a record.

Configuration

@Bean
    public ConcurrentKafkaListenerContainerFactory<String, GGP> kafkaListenerContainerFactory() {

        ConcurrentKafkaListenerContainerFactory<String, GGP> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(10);
        return factory;
    }

What are some considerations to make when setting the following config vals for consumers that can take multiple minutes to complete the consumption of a record.

heartbeat.interval.ms # default 3 seconds
session.timeout.ms # default 45 seconds
request.timeout.ms # default 30 seconds
max.poll.interval.ms # default 5 mins

As I currently understand max.poll.interval.ms should be longer than the processing time of a single record but by how much (is there a good rule of thumb)? I understand that the larger this value the longer it will take to recover from an actual failure. However if this value is too low then it will fail prematurely. Do any of the other configurations need to change because of such a long processing time?

Answer 1

max.poll.interval.ms is the max time between polls from the consumer. It should be set to a value that is longer than the processing time for all the records fetched during the poll ( max.poll.records ). Note that it will also delay group rebalances since the consumer will only join the rebalance inside the call to poll.

A consumer failure is determined by the heartbeat sent by the consumer. The interval for the heartbeat is configured using heartbeat.interval.ms

When the consumer doesnot send the heartbeat for session.timeout.ms it is considered as failed.

Ideally for your usecase,

• session.timeout.ms should be set to a low value to detect failure more quickly. But greater than heartbeat.interval.ms
• max.poll.interval.ms should be set to a large value that is enough to process the max.poll.records .

Note,

The session.timeout.ms value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms .