pass a command line argument to jvm(java) mapper task

Question

I want to debug some parts of my mapper for which I need to pass some command line arguments to the jvm(java) process which starts the mapper. What are the different ways to do this?

I figured out one way to change MapTaskRunner.java, but I want to avoid compiling the whole hadoop package. There should be some simple way using a configuration file to pass extra command line arguments to the jvm mapper process.

Answer 1

I guess you are looking for the following configuration in mapred-config.xml:

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx4096m -XX:+UseConcMarkSweepGC</value>
</property>

In value part you may set your arguments using -D .

Alternatively you may set the HADOOP_OPTS in your terminal:

export HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5000"

For more info on local debugging hadoop jobs, see here .