how to run two zookeeper server in one node

Question

I have three physical nodes with Docker installed on each of them. I have configured Marathon, Flink, Mesos, Zookeeper and Hadoop on each docker. They work really well. I have to distribute data to Flink cluster, so I need Kafka. Zookeeper has been already run; so,Kafka is run without error. The problem is that in this situation, when I want to create Kafka Topic, I see this error which I think it is because I do not run Zookeeper that is in Kafka folder:

Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:230) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:226) at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:95) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1580) at kafka.admin.TopicCommand$.main(TopicCommand.scala:57) at kafka.admin.TopicCommand.main(TopicCommand.scala)

Also, I change my plan to use Zookeeper in Kafka folder. To do that I configure Zookeeper in Kafka folder with new ports like 2186,2889,3889 . But when I run Zookeeper with this command:

  /home/kafka_2.11-2.0.0/bin/zookeeper-server-start.sh /home/kafka_2.11-2.0.0/config/zookeeper.properties

I receive this error:

WARN Cannot open channel to 2 at election address /10.32.0.3:3889 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:534)

at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:454)

at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:435) at java.lang.Thread.run(Thread.java:748)

The configuration of zookeeper which is in "/home/zookeeper-3.4.14/conf/zoo.cfg" in first node:

    tickTime=2000
    initLimit=10
    syncLimit=5        
    dataDir=/var/lib/zookeeper/data
    clientPort=2181
    server.1=0.0.0.0:2888:3888
    server.2=10.32.0.3:2888:3888
    server.3=10.32.0.4:2888:3888

The Zookeeper configuration which is in Kafka folder is like this for first node:

     dataDir=/tmp/zookeeper/data
     tickTime=2000
     initLimit=10
     syncLimit=5
     clientPort=2186
     server.1=0.0.0.0:2889:3889
     server.2=10.32.0.3:2889:3889
     server.3=10.32.0.4:2889:3889

Would you please guide me on how to run two Zookeepers in one docker container? By the way, I cannot use another container for a Kafka cluster, since, I need two containers with one common IP address.

Any help would be really appreciated.

Answer 1

Problem solved. I used above configuration, but used dataDir=/var/lib/zookeeper/data for both Zookeeper. Also, first, I ran Hadoop and then ran Kafka with Zookeeper.