Spring 具有多个分区和多个侦听器的 Kafka

Question

I want to setup multiple partition for a topic, Let say 3 partition. Also it would have 3 consumers within same group reading from these partition (1-1 mapping as per kafka design).

My configurations are as follows:

@Bean
public ConcurrentMessageListenerContainer<String, String> kafkaMessageContainer() throws Exception {
    
    //Setting 3 Partition
    ContainerProperties containerProps = new ContainerProperties(new TopicPartitionOffset(topic, 0), new TopicPartitionOffset(topic, 1), new TopicPartitionOffset(topic, 2));
    //Group id is set same for multiple instances of this service
    containerProps.setGroupId(groupId);
    
    ConcurrentMessageListenerContainer<String, String> factory = new ConcurrentMessageListenerContainer <String, String>(kafkaConsumerFactory(kafkaBootStrapServers), containerProps);
    
    factory.setErrorHandler(new SeekToCurrentErrorHandler(new FixedBackOff(KAFKA_RETRY_INTERVAL, KAFKA_RETRY_MAX_ATTEMPT)));
    //Setting 3 listeners
    factory.setConcurrency(3);
    
    return factory;
}
private ConsumerFactory<String, String> kafkaConsumerFactory(String kafkaBootStrapServers) {
        
    Map<String, Object> config = new HashMap<String, Object>();
    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootStrapServers);
    config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
    
    return new DefaultKafkaConsumerFactory<String, String>(config);
}

When i run this from single instance of my service then listeners are running fine with reading message from their respective topic-partition (no duplicate). But when i created 2 instance of the service then listener from both the instances are reading from same topic-partition again. This leads to duplicate processing of same message.

My question is if i didn't change group id of listener in both instances then why listeners in different instance reading same message again. Also how can i ensure it works fine as in single instance. Thanks

Answer 1

ContainerProperties containerProps = new ContainerProperties(new TopicPartitionOffset(topic, 0), new TopicPartitionOffset(topic, 1), new TopicPartitionOffset(topic, 2));

You are manually assigning the partitions so the group.id is irrelevant for assignment purposes.

If you want to distribute the partitions across multiple instances using group management, you need to just use ContainerProperties containerProps = new ContainerProperties(topic); .

You will need at least 6 partitions for 2 instances with concurrency = 3; otherwise you will have idle consumers.

When there is only one instance, it will get all 6 partitions.