Multiple Cluster Kafka Setup

Question

I would like to setup a multiple kafka cluster with around 3 zookeeper instance, with 3 kafka brokers in each cluster and each kafka broker with around 5 topics and 5 partitions. Is there any setup guide i can refer to?

PS: i can find information for a Single zookeeper instance with multiple Kafka brokers, but not the setups with multiple zookeeper instances.

Answer 1

You should be able to find a beginner's guide, but we found the Apache Kafka documentation contains all answers to questions you might have. In general, we experienced the Kafka set up process painless.

The amount of Zookeeper instances should be odd and >= 3. With 1 Zookeeper instance you have a single point of failure.

By the way, 5 partitions per topic seems quite low. You can always increase the number of partitions. A higher number of partitions enables using more consumers, but also requires more memory usage of the consumer processes, because consumers buffer a certain number of Kafka messages per partition they own. For reference, we use have a similar setup and we use 60 partitions per topic.