Less partitions than consumers in kafka, how to Optimize other consumers of that consumer group

Question

Let's assume the scenario where I have four partitions and a consumer group having twenty consumers. So only four consumers will be working and sixteen will be in starvation mode. How to optimize that sixteen other consumers so we will have every consumer in working, or how we can optimize this situation?

Answer 1

This is desired behaviour. Kafka consumers guarantee that each record in a given partition will be consumed in order; if a partition were to have more than one consumer for a given group, this would no longer hold up.

Additionally, consumer groups can keep check of their progress through any given partition on a topic. If multiple consumers for the same group were competing, recording the offset would become nonsensical.

I guess you'd want to size your consumer groups when you spin up your application's nodes, but remember that the extra consumers in the group will add resiliency as well (maybe a little more than you bargained for in this case!)