简体繁体 English

一名卡夫卡消费者使用多个主题，而每个主题/分区使用一个消费者

[英]One kafka consumer for multiple topics vs one consumer for each topic/partition

原文 2018-03-13 09:43:35 7 1 apache-kafka

I am working on data ingestion use case where data comes on multiple topics and had to be pushed to multiple tables based on the topic name. 我正在研究数据摄取用例，其中数据来自多个 主题，并且必须根据主题名称将其推送到多个表。 I was trying to understand will having one consumer for all the topics has any performance difference with having one consumer for each topic/partition. 我试图理解将所有主题的 一个消费者与每个主题/分区的一个消费者 具有任何 性能差异 。

1 个解决方案

The performance difference between these 2 scenarios depends on the numbers of brokers, partitions and on the expected throughput. 这两种方案之间的性能差异取决于代理，分区的数量以及预期的吞吐量。

When the number of brokers, partitions and throughput is high, if you only have a single consumer for all partitions it's very likely it won't be able to cope with all the traffic. 当代理，分区和吞吐量很高时，如果所有分区只有一个使用者，那么很可能无法应付所有流量。

For example, if you have 5 brokers with 5 partitions on each and each partitions has 5MB/s traffic: 例如，如果您有5个代理，每个代理具有5个分区，并且每个分区的流量为5MB / s：

if you have a single consumer: it will need to have a connection to each broker and will have to handle 5 x 5 x 5 MB/s = 125MB/s. 如果您只有一个使用者：它将需要与每个代理建立连接，并且必须处理5 x 5 x 5 MB / s = 125MB / s。 Depending on your consumer config this might not be feasable. 根据您的使用者配置，这可能不可行。 Even if you can handle 125MB/s, this does not scale well if you end up adding more partitions. 即使您可以处理125MB / s的速度，但如果最终添加更多的分区，扩展性也不会很好。
if you have multiple consumers: each consumer will grab a subset of the partitions. 如果您有多个使用者：每个使用者将获取一部分分区。 With 5 consumers, each will only have to handle 25MB/s which is easily feasable with a standard VM. 如果有5个使用者，则每个使用者仅需处理25MB / s的速度，而使用标准VM则很容易实现。

Kafka's consumer group feature makes it very easy to add consumers on the fly. Kafka的消费者群体功能使您可以轻松地动态添加消费者。 So you can start with only a single consumer and add more if/when the throughput increases. 因此，当吞吐量增加时，您可以仅从单个使用者开始，并添加更多。