單spring boot kafka下的兩個Consumer組和兩個Topic

Question

我是 spring boot kafka 的新手，並試圖將我的 spring-boot kafka 應用程序 (ms-consumer-app) 與與兩個不同消費者組 ID 關聯的兩個不同主題連接起來，兩者都需要在 ms-consumer-app (Spring引導卡夫卡）。

代碼摘要

1. 我有兩個 consumerFactory bean 方法 - consumerFactoryG1 with myGroupId-1 和 consumerFactoryG2 with myGroupId-2
2. 兩個方法用@KafkaListener 注釋為其相應的主題，groupId 和 containerFactory
3. 關聯：容器工廠 = "consumerFactoryG1", 主題 = "test-G1", groupId = "myGroupId-1" 和容器工廠 = "consumerFactoryG2", 主題 = "test-G2", groupId = "myGroupId-2"

但是，當我啟動 ms-consumer-app（Spring boot kafka）時，我得到“org.apache.kafka.common.errors.TopicAuthorizationException”

ERROR [ms-consumer-app] --- [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] org.apache.kafka.clients.Metadata.checkUnauthorizedTopics - [Consumer clientId=consumer-myGroupId-1-3, groupId=myGroupId-1] Topic authorization failed for topics [test-G1]
ERROR [ms-consumer-app] --- [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.error - Authorization Exception and no authorizationExceptionRetryInterval set
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-G1]

我的 KafkaConsumerConfig 類

@Configuration
@EnableKafka
public class KafkaConsumerConfig {
    @Bean
    public ConsumerFactory<String, String> consumerFactoryG1() {
        Map<String, Object> config = new HashMap<>();
        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.GROUP_ID_CONFIG, "myGroupId-1");
        config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(), new StringDeserializer());
    }
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactoryG1() {
        ConcurrentKafkaListenerContainerFactory<String, String> concurrentKafkaListenerContainerFactory = new ConcurrentKafkaListenerContainerFactory<>();
        concurrentKafkaListenerContainerFactory.setConsumerFactory(consumerFactoryG1());
        concurrentKafkaListenerContainerFactory.setMissingTopicsFatal(false);
        return concurrentKafkaListenerContainerFactory;
    }
    
    @Bean
    public ConsumerFactory<String, String> consumerFactoryG2() {
        Map<String, Object> config = new HashMap<>();
        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.GROUP_ID_CONFIG, "myGroupId-2");
        config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(), new StringDeserializer());
    }
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactoryG2() {
        ConcurrentKafkaListenerContainerFactory<String, String> concurrentKafkaListenerContainerFactory = new ConcurrentKafkaListenerContainerFactory<>();
        concurrentKafkaListenerContainerFactory.setConsumerFactory(consumerFactoryG2());
        concurrentKafkaListenerContainerFactory.setMissingTopicsFatal(false);
        return concurrentKafkaListenerContainerFactory;
    }
    
     @KafkaListener(topics = "test-G1", groupId = "myGroupId-1", containerFactory = "kafkaListenerContainerFactoryG1")
        public void getTopicsG1(@RequestBody String emp) {
        System.out.println("Kafka event consumed is: " + emp);
    }
         @KafkaListener(topics = "test-G2", groupId = "myGroupId-2", containerFactory = "kafkaListenerContainerFactoryG2")
        public void getTopicsG2(@RequestBody String emp) {
        System.out.println("Kafka event consumed is: " + emp);
    }
}

任何線索都會有更多幫助。

提前致謝

Answer 1

由於您的配置幾乎相同，因此您不需要兩個工廠； 您可以使用 Spring Boot 的自動配置工廠（通過 application.properties 或 .yml）。 您已經在@KafkaListener上指定了groupId ，這會覆蓋工廠group.id 。

不過，這與您的錯誤無關； 您的經紀人必須配置了安全性，並且您不得使用這些主題。

有關授權的信息，請參閱 Kafka 文檔： https : //kafka.apache.org/documentation/#security_authz