Kafka多个消费者正在收听多个主题

Question

We are working on an application where we want to listen for more than one topics in Kafka. All the topics have a single partition. All the topic names will have a common prefix eg "test-x", "test-y" so we can use spring topicPattern for it.

We wanted to write a java spring consumer which listens to all the topics using regexp pattern. Our idea was, we can run multiple instances of the same consumer(belonging to the same group) and Kafka will distribute messages from different topics for different consumers.

However, This seems to be not working. Even if we are running multiple instances of the consumer only one consumer is getting messages from all the topics. If we create more than one partition in the topic then it works but that's something we won't have.

Is there anything which we have not understood here? Thanks in advance.

Answer 1

Kafka distributes partitions within a topic among a consumer group(consumers with same group.id). Since your topic has only partition, only one of the consumer in a consumer group would receive the message.