可以在Mapper中使用Configuration.set吗？

Question

I am trying to save some data from the Mapper to the Job/Main so that I can use it in other jobs.

I tried to use a static variable in my main class (that contains the main function) but when the Mapper adds data to the static variable and I try to print the variable when the job is done I find that there is no new data, it's like the Mapper modified another instance of that static variable..

Now i'm trying to use the Configuration to set the data from the Mapper:

Mapper

context.getConfiguration().set("3", "somedata");

Main

boolean step1Completed = step1.waitForCompletion(true);
System.out.println(step1.getConfiguration().get("3"));

Unfortunately this prints null .

Is there another way to do things? I am trying to save some data so that I use it in other jobs and I find using a file just for that a bit extreme since the data is only an index of int,string to map some titles that I will need in my last job.

Answer 1

It is not possible as soon as I know. Mappers and Reducers work independently in distributed fashion. Each task has its own local conf instance. You have to persist data to HDFS while each job is independent.

You can also take advantage of MapReduce Chaining mechanism( example ) to run a chain of jobs. In addition, you can design workflow in Azkaban, Oozie and etc to pass output to another job.

Answer 2

It is indeed not possible since the configuration goes from the job to the mapper/reducer and not the other way around. I ended up just reading the file directly from the HDFS in my last job's setup.

Thank you all for the input.