[英]Less partitions than consumers in kafka, how to Optimize other consumers of that consumer group
Let's assume the scenario where I have four partitions and a consumer group having twenty consumers.让我们假设我有四个分区和一个有 20 个消费者的消费者组的场景。 So only four consumers will be working and sixteen will be in starvation mode.
所以只有四个消费者在工作,十六个将处于饥饿模式。 How to optimize that sixteen other consumers so we will have every consumer in working, or how we can optimize this situation?
如何优化其他 16 个消费者,让每个消费者都在工作,或者我们如何优化这种情况?
This is desired behaviour.这是理想的行为。 Kafka consumers guarantee that each record in a given partition will be consumed in order;
Kafka 消费者保证给定分区中的每条记录都会按顺序消费; if a partition were to have more than one consumer for a given group, this would no longer hold up.
如果一个分区对一个给定的组有多个消费者,这将不再成立。
Additionally, consumer groups can keep check of their progress through any given partition on a topic.此外,消费者组可以通过主题上的任何给定分区来检查他们的进度。 If multiple consumers for the same group were competing, recording the offset would become nonsensical.
如果同一组的多个消费者在竞争,记录偏移量将变得毫无意义。
I guess you'd want to size your consumer groups when you spin up your application's nodes, but remember that the extra consumers in the group will add resiliency as well (maybe a little more than you bargained for in this case!)我想您会在启动应用程序节点时调整消费者组的大小,但请记住,组中的额外消费者也会增加弹性(在这种情况下,可能比您讨价还价的要多一点!)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.