我无法在独立模式下配置的hadoop上执行map-reduce作业

Question

I am trying to test a very simple hadoop map-reduce job on my computer (MacOS 10.7) on the local filesystem (in standalone mode). The job takes a .csv file (data-01) and counts the occurrences of some fields.

I downloaded CDH4 hadoop, ran the job, it seemed to start normally but after all the split were processed I got the following error:

13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:9999220736+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000299_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10032775168+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000300_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10066329600+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000301_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10099884032+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000302_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10133438464+32025555
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:19 INFO mapred.LocalJobRunner: Map task executor complete.
13/03/12 12:11:19 WARN mapred.LocalJobRunner: job_local2133287029_0001
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:399)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:389)
    at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:78)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:668)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
13/03/12 12:11:19 INFO mapreduce.Job: Job job_local2133287029_0001 failed with state FAILED due to: NA
13/03/12 12:11:19 INFO mapreduce.Job: Counters: 0

I get the same error no matter how small the input file is...

Answer 1

It happened that the default options were superseding my local configuration (I still don't understand why).

export HADOOP_CLIENT_OPTS="-Xmx1024m"

solved the problem.