Is it possible to run MapReduce locally, without HDFS and Hadoop cluster?

Question

Given I develop MapReduce tasks in my Windows system and before moving them to HDFS cluster I would like to run MapReduce locally. I just want to check how do work my mapper logic, inputSplits, Input/Output formats etc. Is it possible?

Answer 1

Hadoop runs in 3 modes.

1.local mode  
2.psuedo mode  
3.distributed mode. 

The one you are looking for is the local mode. You can debug your mapreduce code from eclipse before run it in 2 or 3 modes.
This step by step guide to run application in local mode helps you a lot to debug your application.
Hope it helps!

Answer 2

In Windows to execute and test the Map Reduce code we need to take the help of cygwin (A windows to Unix simulator).

Steps to follow: 1.First install cywin. 2.set the cygpath environmental variable

now we can give local files as input to MR program.

Answer 3

我们可以通过以下命令在本地进行测试：

cat ./*.csv | python pythonProgram.py | python mapper.py | python reducer.py