不同 Kafka 主题上的分区是否位于同一消费者（k8s pod）中

Question

I have a requirement where I want to be able to read data from partition 1 of topic A and partition 1 of topic B from the same consumer, I have a group of consumers running in different Kubernetes pods. Both topics will have 5 partitions each and both topics have key based partition strategy. So assuming partition 1 on topic A and partition 1 on topic B are keyed with same key value would they both colocate on the same consumer or pod. If that's the case then I can cross reference data from one topic using the key of the other topic's message.

Thank you

Answer 1

There is no guarantee that a consumer will be assigned the same partitions across two topics. Keys are only relevant to the producer partitioner.

If you must consume the same partition of multiple topics, you may assign() those values to the consumer instance rather than subscribe() -ing to the whole topic.

However, if you are wanting to join data across topics, the more appropriate way to do this would be to use Kafka Streams / KSQL joins.

Answer 2

Yes, if you configure routing by key for both topics, same key will be sent to same partition. Have a look at the documentation here: https://kafka.apache.org/documentation/#design_loadbalancing

"For example if the key chosen was a user id then all data for a given user would be sent to the same partition. This in turn will allow consumers to make locality assumptions about their consumption. This style of partitioning is explicitly designed to allow locality-sensitive processing in consumers."