Kafka分区的含义

Question

When we decide about partitions, should we do that on per-topic base, or it is topic-wide decision? If T1 partitioned on 3 partitions, and T2 partitioned on 2 partitions, can they both be consumed by 1 consumer?

Or it is better make equal number of partitions if topics must be consumed by 1 consumer?

I ask that because high level consumer can be created by passing topics and partition number.

So I wonder should I pass to that constructor only topics with equal partition number?

Answer 1

When we create high level consumer, we pass not partition number, but intended number of consuming threads(streams).

The answer is yes, they can be consumed by 1 consumer. (If that consumer subscribed to both topics) Consumer just opens N streams/intended number of consuming threads (you pass that as a parameter!). If N < P(number of all partitions of all topics), then some streams will collect data of several partitions. If N > P, some streams will be in non-busy wait.

It is desirable to have P=N, but it is even better to have N > P , because tomorrow if new partitions appear - you will be ready for grater load.

I've done a research on that and created a blog entry