Hadoop: java.net.ConnectException: Connection refused

Question

Hello I have been trying to follow this tutorial: http://www.tutorialspoint.com/apache_flume/fetching_twitter_data.htm for a very long time now and I am absolutely stuck at Step 3: Create a Directory in HDFS. I have ran start-dfs.sh and start-yarn.sh and both seem to have worked correctly as I am getting the same output as the tutorial but when I try to run:

hdfs dfs -mkdir hdfs://localhost:9000/user/Hadoop/twitter_data 

I keep receiving the same error:

mkdir: Call From trz-VirtualBox/10.0.2.15 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

I can not figure out why as I have searched everywhere and tried a number of solutions but can't seem to make progress. I am going to list all of the files that I think could cause this but I could be wrong: My core.site.xml is:

<configuration>
<property>  
<name>hadoop.tmp.dir</name>
<value>/Public/hadoop-2.7.1/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>

My mapred-site.xml is:

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9001</value>
</property>
</configuration>

My hdfs.site.xml is:

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permission</name>
<value>false</value>
</property>
</configuration>

I am running Ubuntu 14.04.4 LTS on virtual box. My ~/.bashrc looks as so:

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop/bin 
export HADOOP_HOME=/usr/local/hadoop/bin
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
#flume
export FLUME_HOME=/usr/local/Flume
export PATH=$PATH:/FLUME_HOME/apache-flume-1.6.0-bin/bin
export CLASSPATH=$CLASSPATH:/FLUME_HOME/apache-flume-1.6.0-bin/lib/*

And finally my /etc/hosts file is set up as so:

127.0.0.1  localhost
10.0.2.15  trz-VirtualBox
10.0.2.15  hadoopmaster


# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

The added hadoopmaster I am currently not using, that was one of my attempts to fix this by trying not to use local host (didn't work). trz-VirtualBox was originally 127.0.1.1 but I read that you should use your real IP address? Neither worked so I am not sure. I posted all of these files because I do not know where the error is. I do not think it is a path issue (I had a lot before I got to this step and was able to resolve them myself) so I am out of ideas. I've been at this for a number of hours now so any help is appreciated. Thank you.

Answer 1

Found my answer by following this tutorial: http://codesfusion.blogspot.in/2013/10/setup-hadoop-2x-220-on-ubuntu.html

And then with these edits: https://stackoverflow.com/a/32041603/3878508

Answer 2

You have to set permissions to the hadoop's directory

sudo chown -R user:pass /hadoop_path/hadoop

Then start the cluster and run jps command to see the DataNode and NameNode process.

Answer 3

I was getting similar error. Upon checking I found that my namenode service was in stopped state. sudo status hadoop-hdfs-namenode - check status of the namenode

if its not in started/running state sudo start hadoop-hdfs-namenode - start namenode service

Do keep in mind that it takes time before name node service becomes fully functional after restart. It reads all the hdfs edits in memory. You can check progress of this in /var/log/hadoop-hdfs/ using command tail -f /var/log/hadoop-hdfs/{Latest log file}