How this input and output for Hadoop works?

Question

I just setup my single node Hadoop cluster on my local machine using this tutorial -

http://ac31004.blogspot.com/2013/10/installing-hadoop-2-on-mac_29.html

In this tutorial I followed this following instructions : try a ls hadoop fs -ls

if you get

ls: `.': No such file or directory

then there is no home directory in the hadoop file system. So

hadoop fs -mkdir /user
hadoop fs -mkdir /user/<username>

where is the name you are logged onto the machine with.

now change to $HADOOP_INSTALL directory and upload a file

hadoop fs -put LICENSE.txt 

When I used this following command :

hadoop jar ./hadoop-mapreduce-examples-2.2.0 wordcount LICENSE.txt out 

It worked but I am not able to find the output folder. So to debug I tried to use another output folder and used :

hadoop jar ./hadoop-mapreduce-examples-2.2.0 wordcount LICENSE.txt  /Applications/hadoop-2.2.0/out2

This worked fine, but the problem is I don`t seethe directory out2 in the application folder.

Then I created a directory 'in' in the /Applications/hadoop-2.2.0 and put README.txt in it and tried to use following command :

hadoop jar ./hadoop-mapreduce-examples-2.2.0.jar wordcount /Applications/hadoop-2.2.0/in/README.txt  /Applications/hadoop-2.2.0/out3

Now I got an error:

ERROR security.UserGroupInformation: PriviledgedActionException as:chinmay (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/Applications/hadoop-2.2.0/in/README.txt

Answer 1

The input and output folders are also on HDFS. Create them there, not on the local filesystem.