Apache Kafka - Load-Balancing between Consumers consuming only from a specific partition

Question

I understand that in Apache Kafka I can write Producer and Partitioner in such a way that messages of TypeA goes to PartitionA and messages of TypeB goes to PartitionB. And I can also write a Consumer/ConsumerGroup in such a way that Consumer/ConsumerGroupA consumes only from PartitionA and Consumer/ConsumerGroupB consumes only from PartitionB using assign().

But what I really want to understand is, is it a good practice at all. Because, from what I understand, this would severely restrict my load-balancing capability and increase complexity at the same time. The reason being, if my messages of TypeA increase and I want to create another Partition to handle the load, say PartitionA2, and I create more Consumer, add both the new and the old Consumer to a ConsumerGroup and make sure they collectively process items from both the new and the old partitions, will I be able to do it?

Answer 1

Using assign doesn't restrict your load-balancing capabilities but just put in your hands all the problems for reassigning partitions when a new consumer comes up or goes down. It's something that you have for free with the subscribe way. Regarding your specific question, when you add a PartitionA2 for sure you can add another consumer which uses assign for being assigned to such partition.

Answer 2

You can use subscribe API to add more consumer instances to a consumer group. With assign API you have to handle rebalancing yourself. Also if your application depends on partitioning strategy(on event ordering) you may not want to change the partitioning. For example let's say you have one partition for user login/logout actions. Now if you change the partitioning to have 2 partitions( one for login and logout) your application can see logout event before a login event for that particular user. Of course you need to see what is typeA and if it is okay to send the typeA events to 2 different partitions.

Answer 3

感谢ppatierno＆mrnakumar的帮助，尽管我的TypeA事件都将是独立的，并且不需要时间排序，但是我必须处理自己的平衡这一事实无疑是令人沮丧的。