Hadoop java mapper -copyFromLocal heap size error

Question

As part of my Java mapper I have a command executes some code on the local node and copies a local output file to the hadoop fs. Unfortunately I'm getting the following output:

Error occurred during initialization of VM

Could not reserve enough space for object heap

I've tried adjusting mapred.map.child.java.opts to -Xmx512M, but unfortunately no luck.

When I ssh into the node, I can run the -copyFromLocal command without any issues. The ouput files are also quite small like around 100kb.

Any help would be greatly appreciated!

Answer 1

An infinite loop in the mapper or reducer can cause Out of memory errors.

I ran into an OoM once when I had a while-loop with iterator.hasNext() as the condition, for the reducer values, and was not calling iterator.next() inside the loop.