How to create multiple Kafka clusters on same L2 network?

Question

I have 12 Kafka nodes on same L2 Network.

I want to create 4 Kafka clusters (each with 3 nodes) on same L2 network.

Is it possible to do this and how? If not then what's the workaround?

Answer 1

A Kafka cluster is basically defined by the Zookeeper data model that its members use. So, if you want some brokers to belong to a given cluster, you have to make them point to the same Zookeeper namespace. In practical terms, if you want to have 4 clusters then you will need 4 different chroots in your Zookeeper ensemble and make each cluster use a different one. For example, members of cluster A will use zookeeper.connect=127.0.0.1:2181/cluster/a , members of cluster B will use zookeeper.connect=127.0.0.1:2181/cluster/b and so on. Of course another alternative is to use a completely separated Zookeeper ensembles for each cluster.

From ZooKeeper Programmer's Guide

An optional "chroot" suffix may also be appended to the connection string. This will run the client commands while interpreting all paths relative to this root (similar to the unix chroot command). If used the example would look like: "127.0.0.1:4545/app/a" or "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002/app/a" where the client would be rooted at "/app/a" and all paths would be relative to this root - ie getting/setting/etc... "/foo/bar" would result in operations being run on "/app/a/foo/bar" (from the server perspective). This feature is particularly useful in multi-tenant environments where each user of a particular ZooKeeper service could be rooted differently. This makes re-use much simpler as each user can code his/her application as if it were rooted at "/", while actual location (say /app/a) could be determined at deployment time.