Zookeeper cluster not starting on Docker

Question

I am creating a 3 node Zookeeper cluster and a 3 node Kafka cluster. When I am creating the Zookeeper cluster, only one of the zookeeper nodes is created, while the other two exit with the following error, for each of the other two nodes -

zookeeper-3_1  | 2018-02-13 17:38:22,310 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
zookeeper-3_1  | 2018-02-13 17:38:22,322 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
zookeeper-3_1  | 2018-02-13 17:38:22,323 [myid:] - ERROR [main:ZooKeeperServerMain@66] - Unexpected exception, exiting abnormally
zookeeper-3_1  | java.net.BindException: Address in use
zookeeper-3_1  |    at sun.nio.ch.Net.bind0(Native Method)
zookeeper-3_1  |    at sun.nio.ch.Net.bind(Net.java:433)
zookeeper-3_1  |    at sun.nio.ch.Net.bind(Net.java:425)
zookeeper-3_1  |    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
zookeeper-3_1  |    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
zookeeper-3_1  |    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
zookeeper-3_1  |    at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
zookeeper-3_1  |    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
zookeeper-3_1  |    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
zookeeper-3_1  |    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
zookeeper-3_1  |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
zookeeper-3_1  |    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)

I think this is because all the nodes of Zookeeper are using the port 2181. But in my docker-compose.yml file, I have specified different client ports for each of the nodes as well as different ports for the zookeeper servers.

My docker-compose.yml code is -

version: '2'
services:
  zookeeper-1:
    image: zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host

  zookeeper-2:
    image: zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host  

  zookeeper-3:
    image: zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_CLIENT_PORT: 42181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host    

  kafka1:
    image: abc/kafka:latest
    hostname: kafka1
    network_mode: host
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka2:
    image: abc/kafka:latest
    hostname: kafka2
    network_mode: host
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka3:
    image: abc/kafka:latest
    hostname: kafka3
    network_mode: host
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:39092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

I have also put the Dockerfile for creating the Kafka image

FROM centos:7
MAINTAINER abc@xyz.com

ENV KAFKA_BIN=http://www-eu.apache.org/dist/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz

RUN yum install -y wget java-1.8.0-openjdk \
    && cd /tmp && wget -q $KAFKA_BIN \
    && export K_TAR=/tmp/$(ls kafka* | head -1) \
    && mkdir -p /opt/apache/kafka/ && tar -zxf $K_TAR -C /opt/apache/kafka/ \
    && cd /opt/apache/kafka && ln -s $(ls) current \
    && rm -rf $K_TAR

ENV KAFKA_HOME /opt/apache/kafka/current
ENV PATH $PATH:$KAFKA_HOME/bin

ADD resources /home/kafka

RUN groupadd -r kafka \
    && useradd -r -g kafka kafka \
    && mkdir -p /home/kafka \
    && chown -R kafka:kafka /home/kafka \
    && chmod -R +x /home/kafka/scripts \
    && mkdir -p /var/log/kafka \
    && chown -R kafka:kafka /var/log/kafka \
    && mkdir -p /etc/kafka \
    && chown -R kafka:kafka /etc/kafka

USER kafka

Can someone please help me with this - as to why I am not able to start a 3 node Zookeeper cluster when i run the docker-compose up command? Thank you

Answer 1

You are trying to use the host network and then you can only bind the addresses once. so you might want to use different ports or not use the host network and expose ports where necessary.

You can then address them as zookeeper-1, zookeeper-2 ...

The same should be done for the kafka nodes if they want to access the zookeeper they should be accessed through the zookeeper-1, zookeeper-2 ... (I can not test this as I can not build the abc/kafka image as I miss the resources folder in the Dockerfile)