Create a Kafka brokers cluster using Apache Kafka 0.10.0 API with Java

Question

I want to create a broker cluster using Kafka 0.10 API preferable with Java . As far as I have read kafka_2.11-0.10.0.0.jar do support creating broker using :

import kafka.cluster.Broker;
import kafka.cluster.Cluster; 

But I can't find any documentation for doing so. I recently read [1], which tell how to create a topic using Kafka API in Java . Can we do similar things to create broker cluster, update partition, migrate existing data/partition to new broker (as these new broker will not automatically be assigned any data partitions, so unless partitions are moved to them they won't be doing any work [2])

[1] How Can we create a topic in Kafka from the IDE using API

[2] https://kafka.apache.org/0100/ops.html#basic_ops_cluster_expansion

Answer 1

I have some sample code which you may find useful.

For creating a broker, take a look at KafkaTestServer . It's really for simpler testing, so it does not create a cluster, just a single broker, but it should not be difficult to extend.

Once I built in the ability to create/query/delete topics into the test server, I created a standalone admin client for doing the same against other servers, so if you are already creating a broker cluster, you should be able to use the code to maintain topics on it. Take a look at KafkaAdminClient .

The admin client is basically a pure java wrapper around the scala kafka.admin.AdminUtils class so it handles all the scala <--> java conversions under the covers.