No start of Hadoop components on the slave machine

Question

I am trying to set up a multi-node hadoop cluster using my two laptops using Michael Noll tutorial. The OS on both machines is Ubuntu 14.04.

I managed to set up single-node clusters on each of the two laptops, but when I try to start (after all the necessary modifications as instructed in the tutorial) the multi-node cluster using sbin/start-all.sh on my master the slave does not react at all. All the five components on the master start, but no single one starts on the slave.

My /etc/hosts looks on both PCs like this

127.0.0.1       localhost
192.168.178.01  master
192.168.178.02  slave

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

(Furthermore, in /usr/local/hadoop/etc/hadoop there was no file called master, so I created it using: touch /usr/local/hadoop/etc/hadoop/master )

Then, when I ru sbin/start-all.sh , I see the following:

hduser@master:/usr/local/hadoop$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/05/17 21:21:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-master.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-master.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-master.out
15/05/17 21:21:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-master.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-master.out
hduser@master:/usr/local/hadoop$ jps
3716 DataNode
3915 SecondaryNameNode
4522 Jps
3553 NameNode
4210 NodeManager
4073 ResourceManager
hduser@master:/usr/local/hadoop$ 

Interesting is here that on the line 6 there is localhost . Shouldn't it be master?

I can connect to the slave using ssh slave password-lessly from the master and control the slave machine, but still, sbin/start-all.sh does not start any hadoop components on the slave.

Very intrestingly, if I run sbin/start-all.sh on the slave, it starts NameNode on the master (!!!) and starts NodeManager and ResourceManager on the slave itself.

Can someone help me to properly start the multi-node cluster?

PS: I looked at this , but in my case the location of hadoop home on both machines are identical

Answer 1

There can be several things:

• Check that you can connect with ssh password-less from slave to master. Here is a link that teach us how to do it.
• The hostname of each machine is correct?
• /etc/hosts file is identical on both, master and slave, alike?
• Have you ckecked with ifconfig -a the ip of both machines? Are them the ones that you expected?
• Have you changed all the configurations file in slave machine, so instead of localhost, now must say the master's hostname? You should seek for the words localhost and stuff like that, in all your files on your $HADOOP_HOME directory, because there are several files for configurating all sort of things and it's very easy to forget some. Something like this: sudo grep -Ril "localhost" /usr/local/hadoop/etc/hadoop
• Check the same as before but in master, so instead of saying localhost, it says the hostname of it.
• You should remove the localhost entry, on the /etc/hosts file on slave machine. Sometimes that entry, so typical of the hadoop tutorials, could lead to some problems
• In masters and in slaves @ slave host, it should say only "slave", and in your master host, in masters file it should say "master" and in your slave file, it should say slave.
• You should format your filesystem on both nodes previous to start hadoop.

Those are all the problems that i remember to have when i do the as you are doing right now. Check if some of them help you!