使用hadoop作业的输出作为另一个的输入

Question

I have having a problem using the output of one M/R job as the input of another. According this post , and many other online resources, a way to do this is to create a job1, and then a job2. However, when I do this, I am getting this error:

Error: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.LongWritable

The only time I use LongWritable as a class, is as the key to my mapper. I remember that this needs to stay like this, as this is the offset in the input file. When I change the signature, to be Text , like so:

public class ErrorMapperCombiner extends Mapper<Text, Text, Text, IntWritable>

I get this error:

Error: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

So how can I use the output of one M/R job as the input to another?

I am using this in my "runner" class to chain the too:

job1.setOutputFormatClass(SequenceFileOutputFormat.class);

job1.setOutputKeyClass(Text.class);
job1.setOutputValueClass(IntWritable.class);

.....

job2.setInputFormatClass(SequenceFileInputFormat.class);

Answer 1

I was setting the value twice. (eg)

job1.setOutputFormatClass(TextOutputFormat.class);
....
job1.setOutputFormatClass(SequenceFileOutputFormat.class);

Though the SequenceFileOutputFormat was after it, so I would think that value would be used. But nevertheless, works now.