在Mesos上运行多个Kafka经纪人作为马拉松工作

Question

I want to run multiple Kafka brokers on Mesos as Marathon jobs. I'm thinking what to use as a broker ID. I know that some people are using IP based broker ID, but I read that this approach is not optimal for situations when migrating the brokers to different machines. Any insights?

Related to that, is there a way to do a rolling deploy of Kafka brokers in a way that Marathon waits for the old broker instances to replicate the data to the new ones before killing them.

Answer 1

I created a Docker image which can start on Marathon and also scale up and down. To get a unique broker id, I concatenate the last octet of the IP address and the port number which Marathon devises to the Docker container.

I understood that this shouldn't be a problem, because normally the clients connect via ZooKeeper, which itself has the nodes and their connection info, so you shouldn't need to take care of the details. Or I misunderstood the question...

Have a look at

• http://kafka.apache.org/documentation.html#replication