Multiple Kafka Listeners With Same GroupId All Receiving Message

Question

I have a kafka listener configured in our Spring Boot application as follows:

@KafkaListener(topicPartitions = @TopicPartition(topic = 'data.all', partitions = { "0", "1", "2" }), groupId = "kms")
public void listen(ObjectNode message) throws JsonProcessingException {
    // Code to convert to json string and write to ElasticSearch
}

This application gets deployed to and run on 3 servers and, despite all having the group id of kms , they all get a copy of the message which means I get 3 identical records in Elastic. When I'm running an instance locally, 4 copies get written.

I've confirmed that the producer is only writing 1 message to the topic by checking the count of all messages on the topic before and after the write occurs; it only increases by 1. How can I prevent this?

Answer 1

When you manually assign partitions like that, you are responsible for distributing the partitions across the instances.

The group is ignored.

You must use group management and let Kafka do the partition assignment for you, or assign the partitions manually for each instance.

Instead of topicPartitions , use topics = "data.all"

Answer 2

What happens when you don't assign partition manually

Producer Side

• When producer sends a message without any strategy or specifying to which partition message should go then kafka tries to use round robin technic and splits all messages among all available partition.
  • Message in 2 partitions are unique because maximum of only 1 consumer is recommended to listen to a particular partition of a topic.

Consumer Side

• For example there are 2 partitions of a topic
• Then a consumer(let say A ) joins with consumer group (lets say consumer )
• Partition reassignment happens whenever new consumer joins and 2 partitions get assigned to A as we have only one consumer group consumer
• Now, consumer B tries to join same consumer group consumer then again partition reassignment will happen and both A & B will get partition to listen to messages
• As we have only 2 partitions, even if we add more consumers to same consumer group, there will be only 2 consumer who will be listening to the messages sent to the topic because at a time only 2 consumers can get 1-1 partition. To maintain exclusivity of the message consumed at consumer.

What is happening in your case is, more than 1 consumer is listening to same partitions so all the consumers who are listening to same partitions within same consumer group also, will receive messages from that partition. So mutual exclusivity between consumers in a consumer group is lost due to more than 1 consumer is listening same partitions.