Ubuntu上的Hadoop多节点集群安装问题-故障排除

Question

I have three Ubuntu 12.04 LTS computers that I want to install Hadoop on in a Master/Slave configuration as described here . It says to first install Hadoop as a single node and then proceed to multi-node. The single node installation works perfectly fine. I made the required changes to the /etc/hosts file and configured everything just as the guide says, but when I start the Hadoop cluster on the master, I get an error.

My computers, aptly named ironman, superman and batman, with batman (who else?) being the master node. When I do sudo bin/start-dfs.sh , the following shows up.

When I enter the password, I get this:

When I try sudo bin/start-all.sh , I get this:

I can ssh into the different terminals, but there's something that's not quite right. I checked the logs on superman/slave terminal and it says that it can't connect to batman:54310 and some zzz message. I figured my /etc/hosts is wrong but in fact, it is:

I tried to open port 54310 by changing iptables, but the output screens shown here are after I made the changes. I'm at my wit's end. Please tell me where I'm going wrong. Please do let me know if you need any more information and I will update the question accordingly. Thanks!

UPDATE: Here are my conf files.

core-site.xml Please note that I had put batman:54310 instead of the IP address. I only changed it because I thought I'd make the binding more explicit.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://130.65.153.195:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>130.65.153.195:54311</value>
<description>The host and port that the MapReduce job tracker runs
at.  If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>

</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>

</configuration>

My conf/masters file is simply batman and my conf/slaves file is just:

batman
superman
ironman

Hope this clarifies things.

Answer 1

First things first: Make sure you can ping the master from slave and slave from master. Login to each machine individually and ping the other 2 hosts. Make sure they are reachable via their hostnames. It is possible that you have not add /etc/hosts entries in the slaves.

Secondly, you need to setup passwordless SSH access. You can use ssh-keygen -t rsa and ssh-copy-id for this. This will help remove the password prompts. It is a good idea to create a separate user for this (and not use root).

If this doesn't help, please post your log output.