How consumer groups works in kafka?

Question

Hi I am working on kafka CLI to get clear understanding how kafka works. I am confused on consumer groups. I have created topic with three partitions. I will create producer to feed some data to topic. First time I added some data as below.

kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user1
kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user2
kafka-console-producer --broker-list 127.0.0.1:9092 --topic users 
>user3

Now my understanding is user1,user2,user3 will go randomly to three different partitions.

When creating consumer group as below.

kafka-console-consumer --bootstrap-server localhost:9092 --topic users  --group user_group

This will give me all the user1,user2,user3.

Now inside one consumergroup I can have many consumers. If I have three consumer inside consumer group then first consumer will read from partition1, second consumer will read from consumer2 then third consumer will read from consumer3. This is my understanding so far. If my understanding is correct then what would be the cli command to demonstrate above behavior? I know only one command mentioned above which will return all the data? If my above understanding is correct then If all the consumer requires all the data then how to get it? Can someone help me to understand this concepts. Any help would be greatly appreciated. Thanks

Answer 1

Let's start by understanding the Partitions to Consumers relationship.

Let's say I have a topic called T1 with 4 partitions, and 1 consumer group. In this case, Consumer Group 1 will by assigned to consume from all of the partitions -

Now, when we add another consumer to the same Consumer Group, the partitions will be evenly distributed between them -

And so on when adding another consumers, up to the number of partitions in that topic -

Adding more consumers beyond the amount of partitions in a given topic will cause an idle consumers -

That's basically means that you are bounded to the amount of partitions in a single topic.

How does consumers join Consumer Group?
When a consumer wants to join to a Consumer Group, he sends a JoinGroup request to the group coordinator. The first one to join the group becomes the group leader, and he is the one who is responsible for assigning a subset of the partitions to each consumer, based on a predefined assignment policy.
After deciding on the partition assignment for each consumer, the consumer leader will send the list of assignment partitions to the Group Coordinator and he will send this information to all the consumers within the group.

How to choose the assignment policy?
Kafka supports few assignment policy that can be controlled using the partition.assignment.strategy parameter.
The policies are RangeAssignor , RoundRobinAssignor and StickyAssignor where the default one is RangeAssignor .

You can read more about them on this useful blog post .

How to see it?
I would recommend tools like Kafka Manager that will help you visualize the consumer to topics relations.

Answer 2

You would need to run three console consumers in parallel with the --group option in order to observe the behavior you're expecting.

And the partitions are randomly distributed to the group members, not necessarily that partition 1 goes to the first consumer, and so on. The first and only consumer in a group will always get all partitions, when a second joins, then one of the consumers is reading two partitions