Wordcount example hadoop

Question

I am new to Hadoop and trying to explore it via examples available on their website.So this might sound like a very basic question.

In Eclipse IDE I simply created a java project ,imported necessary hadoop jars and the core jar. I then ran a WordCount example that comes with downloadable and it ran successfully.

However, I didn't setup any Hadoop installation or a single node cluster.

Hence I am unable to figure out how can that job run even without hadoop being configured in my system.

My initial understanding was I will need to install hadoop following something like Running Hadoop on Ubuntu Linux to get going, but apparently that was not necessary. Why?

Answer 1

It's very likely that you run an internal test or something like that (the hadoop configuration can be generated in memory for testing purposes). You should install a single-node cluster and start doing real things, like that one in the guide that you previously referenced. Remember to check oficial hadoop site also.

Answer 2

This probably happens in the scenario where the job only detects the local file system, where it uses the LocalFileSystem API to interact with the files in the local file system.

Please refer the below link, Unit testing hadoop hdfs writings with MiniDFSCluster

This is one of the option to Unit Test the mapreduce/hdfs code that we develop in the dev environment.While you deploy the same code in the hadoop clsuter , the input files will be expected in the HDFS location.