Zookeeper cluster set up

Question

I am able to set up zookeeper cluster on 1 machine with 3 different ports, but when I do the same with different IP to have zookeeper instance on different machines, it throws following error:

2014-11-20 12:16:24,819 [myid:1] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2014-11-20 12:16:24,827 [myid:1] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2014-11-20 12:16:24,842 [myid:1] - INFO  [main:QuorumPeer@959] - tickTime set to 2000
2014-11-20 12:16:24,842 [myid:1] - INFO  [main:QuorumPeer@979] - minSessionTimeout set to -1
2014-11-20 12:16:24,842 [myid:1] - INFO  [main:QuorumPeer@990] - maxSessionTimeout set to -1
2014-11-20 12:16:24,842 [myid:1] - INFO  [main:QuorumPeer@1005] - initLimit set to 10
2014-11-20 12:16:24,857 [myid:1] - INFO  [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: /172.16.1.175:2223
2014-11-20 12:16:24,870 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer@714] - LOOKING
2014-11-20 12:16:24,873 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@815] - New election. My id =  1, proposed zxid=0x0
2014-11-20 12:16:24,876 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2014-11-20 12:16:24,881 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 2 at election address /172.16.1.170:2223
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
    at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
    at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
    at java.lang.Thread.run(Thread.java:744)

Answer 1

have you started zookeeper in all the three nodes ? In a multi-cluster set up (assuming you have a distributed environment with multiple machines) every server knows about the other nodes present in the cluster known as ensemble . It does this by looking at the following piece of line in the zoo.cfg file.

    server.1=zoo1:2888:3888
    server.2=zoo2:2888:3888

In multi-cluster set up doc page it says

As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines

now unless you start the process in all three nodes it wont be able to communicate with each other and keep logging such errors. This probably might help you get somewhere.

Answer 2

How to Setup Zookeeper for Multiple Clusters or Remote servers?

Step 1 : Check the Java 1.8.0 or above version is available in the system under /Opt/ java -version

Step 2 : Download Zookeeper-3.3.6 from the link by using the below command

Sudo wget  http://redrockdigimark.com/apachemirror/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

Step 3 : Extract the File by using the below Command

Sudo tar xzf  zookeeper-3.3.6.tar.gz -C /opt/

Step 4 : Mapper the zookeeper -3.3.6 to Zookeeper as below

/opt/>  ls -s zookeeper-3.3.6 zookeeper         then

/opt/>   Cd zookeeper/conf

Step 5 : Create a Configuration file by copying of zoo.cfg from zoo_sample.cfg /opt/zookeeper/conf/>

    cp zoo.cfg sample_zoo.cfg

Step 6 : Edit the zoo.cfg by using the command /opt/zookeeper/conf/>

sudo vi zoo.cfg

Create the Data directory as DataDir=/var/lib/zookeeper

Step 7 : Create a file without extension as myid under /var/lib/zookeeper and give the unique id as 1 for server1

Add all the cluster server in the botton as

server.1=0.0.0.0:2888:3888
server.2=184.72.205.209:2888:3888
server.3=34.207.92.20:2888:3888

Step 8 : Create a file without extension as myid under /var/lib/zookeeper And give the unique id as 2 for server2

Step 9 : The Same configuration to be applied for the second server as below

server.1=34.229.138.19:2888:3888
server.2=0.0.0.0:2888:3888
server.3=34.207.92.20:2888:3888

Step 10 : Install nc package and lsof packages as below

Sudo yum install nc

Sudo yum install lsof

Step 11 :Now Start the Zookeeper in all servers as

Sudo /opt/zookeeper/bin/zkServer.sh start

Step 12 : To Stop the Zookeeper Server

Sudo /opt/zookeeper/bin/zkServer.sh Stop

To Check the Status of Zookeeper Server

Sudo /opt/zookeeper/bin/zkServer.sh Status

Important Points to be noted

1.For Zookeeper 2F+1 server to be maintained ie. If you have 1 servers then (2*1)+1=3 Servers to be maintained , if you have 2 servers then (2*2)+1=5 Servers to be maintained , F stands for number of servers

2.All the Servers should have zoo.cfg configuration file and the local servers IP should be 0.0.0.0

3.zookeeper uses 2888 port to connect to individual followers nodes with the leader node

4.Port 3888 is for peer to peer communication

5.Leader election will be taken care by zookeeper automatically, and if the leader down, with in 2 micro seconds , it will elect the other leader and shares the information of the followers

6.In zoo.cfg configuration file Client port must be 2181