spark standalone cluster slave unable to connect slave to master

Question

i have the bin for spark-1.6.0-bin-hadoop2.6 im having issue trying to connect the slave to the master so far i have tried(on ubuntu 14.04 live usb):

1. apt-get purge and install openssh-client and server on both systems

2. I have stated explicitly the ip address of the master in the spark url for the worker spark://< master ip>:7077 and also tried changing the SPARK_MASTER_IP in /conf/spark-env.sh the worker executes but the log has the following error im assuming there must be some ssh setup involved but i have tried ssh-keygen and ssh-copy-id @ it also doesnt give any reuslts

    16/02/22 07:49:16 INFO Worker: Connecting to master 192.168.0.208:7077... 16/02/22 07:49:16 WARN Worker: Failed to connect to master 192.168.0.208:7077 java.io.IOException: Failed to connect to /192.168.0.208:7077 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused: /192.168.0.208:7077 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:740) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more 

   16/02/22 07:49:27 INFO Worker: Retrying connection to master (attempt # 2)

i am however able to open the master webUI by typing :8080 on my browser . i am also able to access the webUI of the slave from the master . im almost at the point of giving in so please helllppp.

Answer 1

Make sure that each master and worker has a firewall exception to allow connections for all other workers and masters.

Here's a simplified example from one of our master machines (master0):

$iptables -L

...
ACCEPT     all  --  worker0.company.com  master0.company.com
ACCEPT     all  --  worker1.company.com  master0.company.com
ACCEPT     all  --  master1.company.com  master0.company.com
...

Of course you can also use IPs instead of hostnames.