使用Hadoop，如何更改给定作业的映射器数量？

Question

So, I have two jobs, Job A and Job B. For Job A, I would like to have a maximum of 6 mappers per node. However, Job B is a little different. For Job B, I can only run one mapper per node. The reason for this isn't important -- let's just say this requirement is non-negotiable. I would like to tell Hadoop, "For Job A, schedule a maximum of 6 mappers per node. But for Job B, schedule a maximum of 1 mapper per node." Is this possible at all?

The only solution I can think of is :

1) Have two folders off the main hadoop folder, conf.JobA and conf.JobB. Each folder has its own copy of mapred-site.xml. conf.JobA/mapred-site.xml has a value of 6 for mapred.tasktracker.map.tasks.maximum. conf.JobB/mapred-site.xml has a value of 1 for mapred.tasktracker.map.tasks.maximum.

2) Before I run Job A :

2a) Shut down my tasktrackers

2b) Copy conf.JobA/mapred-site.xml into Hadoop's conf folder, replacing the mapred-site.xml that was already in there

2c) Restart my tasktrackers

2d) Wait for the tasktrackers to finish starting

3) Run Job A

and then do a similar thing when I need to run Job B.

I really don't like this solution; it seems kludgey and failure-prone. Is there a better way to do what I need to do?

Answer 1

In your Java code for the custom jar itself you could set this configuration mapred.tasktracker.map.tasks.maximum for both of your jobs.

Do something like this:

Configuration conf = getConf();

// set number of mappers
conf.setInt("mapred.tasktracker.map.tasks.maximum", 4);

Job job = new Job(conf);

job.setJarByClass(MyMapRed.class);
job.setJobName(JOB_NAME);

job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(MapJob.class);
job.setMapOutputKeyClass(Text.class);
job.setReducerClass(ReduceJob.class);
job.setMapOutputValueClass(Text.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.setInputPaths(job, args[0]);

boolean success = job.waitForCompletion(true);
return success ? 0 : 1;

EDIT :

You also need to set the property mapred.map.tasks to the value derived from the following formula ( mapred.tasktracker.map.tasks.maximum * Number of tasktracker Nodes in your cluster) .