同一消费者组内具有不同分区数量的不同 kafka 主题

Question

Can I have different kafka topic with different amount of partitions within the same consumer group? If so, can I scale the consumers to the number of the topic with the most partitions? Thanks

Answer 1

Yes, each topic can independently have any number of partitions. I'm assuming you are consuming multiple topics with the same consumer, in any case, what will happen is that the topics that have fewer partitions than consumers will be consumed by a subset of the consumers (some consumers will not get any messages from these topics since they won't have any partition assigned)