简体繁体 English

为什么在kafka中对两个Kstream进行共分区，两个流都需要相同数量的分区？

[英]Why does co-partitioning of two Kstreams in kafka require same number of partitions for both the streams?

原文 2017-08-07 10:54:02 9 1 apache-kafka/ apache-kafka-streams

我想知道为什么在kafka中对两个Kstream进行共分区，为什么两个流都需要相同数量的分区，如下面URL中的文档所示：在此处输入链接描述

1 个解决方案

As the name "co-partition" indicates, you want to put data from different topic but same key to the same Kafka Streams application instance. 正如名称“ co-partition”所指示的那样，您要将来自不同主题但具有相同密钥的数据放入同一Kafka Streams应用程序实例。 If you don't have the same number of partitions, it's not possible to get this behavior. 如果您没有相同数量的分区，则无法获得此行为。

Assume you have topic A with 2 partitions and topic B with 3 partitions. 假设您的主题A具有2个分区，主题B具有3个分区。 Thus, it can happen that one record with key X is hashed to partitions A-0 and B-1 (ie, not same partition number). 因此，可能发生的情况是，具有键X的一条记录被哈希到分区A-0和B-1（即，不同的分区号）。 However, for a different key Y it might be hashed to A-0 but B-2. 但是，对于其他密钥Y，可能会将其哈希到A-0，但会哈希到B-2。

Only if the number of partitions is the same for both topics, records with same key end up in the same partitions (of different topics of course), and this allows to process A-0/B-0 and A-1/B-1 etc together. 仅当两个主题的分区数相同时，具有相同键的记录才最终位于相同的分区（当然是不同主题）中，这允许处理A-0 / B-0和A-1 / B- 1等在一起。

加入两个 KStreams 时需要 Kafka Streams Co-Partitioning - Kafka Streams Co-Partitioning is required while joining two KStreams

合并两个 kafka 流是否保留共同分区 - Does Merging two kafka-streams preserve co-partitioning

Kafka 流联合分区与交互式查询 - Kafka streams co-partitioning vs interactive query

Kafka Streams：混合匹配的PAPI和DSL KTable不进行共分区 - Kafka Streams: mix-and-match PAPI and DSL KTable not co-partitioning

Kafka Streams GlobalKTable 主题是否需要与它将加入的 KStream 主题相同数量的分区？ - Does Kafka Streams GlobalKTable topic require the same number of partitions as KStream topic which it will be joining with?

共同分区如何确保来自2个不同主题的分区最终分配给同一Kafka Stream Task？ - How do co-partitioning ensure that partition from 2 different topics end up assigned to the same Kafka Stream Task?

kafka 流中全局状态存储的分区数 - number of partitions for global state store in kafka streams

ksqldb 流是否要求两个模式具有相同数量的字段？ - does ksqldb streams require both schemas to have the same amount of fields?

Kafka-streams：为什么所有分区都分配给使用者组中的同一使用者？ - Kafka-streams: Why do all partitions get assigned to the same consumer in the consumergroup?

Kafka，不同的分区将具有相同的偏移号 - Kafka, will different partitions have the same offset number

暂无

暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 加入两个 KStreams 时需要 Kafka Streams Co-Partitioning - Kafka Streams Co-Partitioning is required while joining two KStreams 合并两个 kafka 流是否保留共同分区 - Does Merging two kafka-streams preserve co-partitioning Kafka 流联合分区与交互式查询 - Kafka streams co-partitioning vs interactive query Kafka Streams：混合匹配的PAPI和DSL KTable不进行共分区 - Kafka Streams: mix-and-match PAPI and DSL KTable not co-partitioning Kafka Streams GlobalKTable 主题是否需要与它将加入的 KStream 主题相同数量的分区？ - Does Kafka Streams GlobalKTable topic require the same number of partitions as KStream topic which it will be joining with? 共同分区如何确保来自2个不同主题的分区最终分配给同一Kafka Stream Task？ - How do co-partitioning ensure that partition from 2 different topics end up assigned to the same Kafka Stream Task? kafka 流中全局状态存储的分区数 - number of partitions for global state store in kafka streams ksqldb 流是否要求两个模式具有相同数量的字段？ - does ksqldb streams require both schemas to have the same amount of fields? Kafka-streams：为什么所有分区都分配给使用者组中的同一使用者？ - Kafka-streams: Why do all partitions get assigned to the same consumer in the consumergroup? Kafka，不同的分区将具有相同的偏移号 - Kafka, will different partitions have the same offset number

相关标签

粤ICP备18138465号 © 2020-2024 STACKOOM.COM