Spring-Kafka Consumer Group Coordination for ConcurrentKafkaListenerContainerFactory

Question

I have couple of questions regarding the behaviour of spring-kafka during certain scenarios. Any answers or pointers would be great.

Background: I am building a kafka consumer which talk with external apis and sends acknowledge back. My Config looks like this:

@Bean
public Map<String, Object> consumerConfigs() {
    Map<String, Object> props = new HashMap<>();

    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerServers());
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, this.configuration.getString("kafka-generic.consumer.group.id"));
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "5000000");
    props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "6000000");

    return props;
}


@Bean
public RetryTemplate retryTemplate() {
    final ExponentialRandomBackOffPolicy backOffPolicy = new ExponentialRandomBackOffPolicy();
    backOffPolicy.setInitialInterval(this.configuration.getLong("retry-exp-backoff-init-interval"));
    final SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
    retryPolicy.setMaxAttempts(this.configuration.getInt("retry-max-attempts"));
    final RetryTemplate retryTemplate = new RetryTemplate();
    retryTemplate.setBackOffPolicy(backOffPolicy);
    retryTemplate.setRetryPolicy(retryPolicy);
    return retryTemplate;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Event> retryKafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, Event> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());

    factory.setConcurrency(this.configuration.getInt("kafka-concurrency"));
    factory.setRetryTemplate(retryTemplate());
    factory.getContainerProperties().setIdleEventInterval(this.configuration.getLong("kafka-rtm-idle-time"));
    //factory.getContainerProperties().setAckOnError(false);
    factory.getContainerProperties().setErrorHandler(kafkaConsumerErrorHandler);
    factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE);
    return factory;
}

Lets say no of partitions I have is 4. My partition distribution is for KafkaListener is:

@KafkaListener(topicPartitions = @TopicPartition(topic = "topic", partitions = {"0", "1"}),
            containerFactory = "retryKafkaListenerContainerFactory")
public void receive(Event event, Acknowledgment acknowledgment) throws Exception {
    serviceInvoker.callService(event);
    acknowledgment.acknowledge();
}

@KafkaListener(topicPartitions = @TopicPartition(topic = "topic", partitions = {"2", "3"}),
        containerFactory = "retryKafkaListenerContainerFactory")
public void receive1(Event event, Acknowledgment acknowledgment) throws Exception {
    serviceInvoker.callService(event);
    acknowledgment.acknowledge();
}

Now my questions are:

1. Let's say I have 2 machines where I deployed this code (with the same consumer group id). If I understood properly, if I get a event for a partition, one of the machines' kafkalistener for corresponding partition and will listen but the other machines' kafkalistener won't listen to this event. Is it?

2. My error handler is:

  @Named public class KafkaConsumerErrorHandler implements ErrorHandler { @Inject private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry; @Override public void handle(Exception e, ConsumerRecord<?, ?> consumerRecord) { System.out.println("Shutting down all the containers"); kafkaListenerEndpointRegistry.stop(); } } 

Lets talk abt a scenario where a consumers' kafkalistener is called where it calls serviceInvoker.callService(event); but the service is down, then according to the retryKafkaListenerContainerFactory , it retries for 3 times then fails, then errorhandler is called thus stopping kafkaListenerEndpointRegistry. Will this shutdown all other consumers or machines with the same consumer group or just this consumer or machine?

3. Lets talk abt the scenerio in 2. Is there any configuration where we need to change to let kafka know to hold off acknowledgement for that much time?

4. My kafka producer produces messages for every 10 mins. Do I need to configure that 10 mins anywhere in my Consumer code or is it agnostic of such?

5. In my KafkaListener annotations I hardcoded topic name and partitions. Can I change it during run time?

Any help is really appreciated. Thanks in advance. :)

Answer 1

1. Correct; only 1 will get it.
2. It will only stop the local containers - Spring doesn't know anything about your other instances.
3. Since you have ackOnError=false , the offset won't be committed.
4. The consumer does not need to know how often messages are published.
5. You can't change them at runtime, but you can use property placeholders ${...} or Spel Expressions #{...} to set them up during application initialization.