Kafka Consumer distribution not working as expected

Question

I have Three topics each having three partitions on a cluster of kafka.

now, there are total 9 partitions. and when i create 9 consumers... the 6 are being idle. only three consumers are being used.

the expectation is: each consumer should pickup one partitions and hence, 9 consumer should pick up documents from 9 partitions

but what happens is: one consumer picks up messages from three paritions one of different topic.

eg i have three topics Topic_A,Topic_B and Topic_C and three partitions each. hence parititions are as below:

Topic_A_0, Topic_A_1, Topic_A_2, Topic_B_0, Topic_B_1, Topic_B_2, Topic_C_0, Topic_C_1, Topic_C_2

When i create 9 consumers, the distribution works as below:

Consumer1: Topic_A_0,Topic_B_0,Topic_C_0 
Consumer2: Topic_A_1,Topic_B_1,Topic_C_1 
Consumer3: Topic_A_2,Topic_B_2,Topic_C_2
Consumer4,Consumer5,Consumer6,Consumer7,Consumer8,Consumer9 are idle

It should be

Consumer1: Topic_A_0
Consumer2: Topic_A_1
Consumer3: Topic_A_2

Consumer4: Topic_B_0
Consumer5: Topic_B_1
Consumer6: Topic_B_2

Consumer7: Topic_C_0 
Consumer8: Topic_C_1 
Consumer9: Topic_C_2

Is there any configuration i need to let all 9 consumer pick up messages from 9 unique parititons?

Answer 1

Make sure your all your consumers are subscribing to same set of topics under the same consumer group id. For the list of topics, you can pass a predefined list or a regular expression for consumers to subscribe from. The consumer-id can be set using group.id property in consumer.

Answer 2

The default partition assignment strategy doesn't work across topics so this is the expected behaviour. A similar question here : Kafka Consumers are balanced across topics