ChainMapper泛型的Java Hadoop问题

Question

Hi I am currently facing an issue with Java Generics: The method I need to use has this signature

static<K1,V1,K2,V2> void addMapper(JobConf job, 
           Class<? extends Mapper<K1,V1,K2,V2>> klass, 
           Class<? extends K1> inputKeyClass, 
           Class<? extends V1> inputValueClass, 
           Class<? extends K2> outputKeyClass, 
           Class<? extends V2> outputValueClass, 
           boolean byValue, JobConf mapperConf) 

and this is how I call it

ChainMapper.addMapper(conf, NameMapper.class, 
           Object.class, Object.class, Object.class, Object.class, 
           false, nameYearConf);

where NameMapper is defined as follows

public class NameMapper extends Mapper<Object, Object, Object, Object> { }

and also the other two parameters are correctly intialized

JobConf conf = new JobConf(); JobConf nameYearConf = new JobConf(false);

when I try to compile I obtain the following error

    src\MeanYear.java:107: error: method addMapper in class ChainMapper cannot be applied to given types;
                    ChainMapper.addMapper(conf, NameMapper.class, Object.class,
                               ^
      required: JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf
      found: JobConf,Class<MeanYear.NameMapper>,Class<Object>,Class<Object>,Class<Object>,Class<Object>,boolean,JobConf
      reason: no instance(s) of type variable(s) K1,V1,K2,V2 exist so that argument type Class<MeanYear.NameMapper> conforms to formal parameter type Class<? extends Mapper<K1,V1,K2,V2>>
      where K1,V1,K2,V2 are type-variables:
        K1 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
        V1 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
        K2 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
        V2 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)

I really cannot understand what I am doing wrong, I tried some similar examples and the compiles and they work. I have also tried some examples provided here, but without success. Thanks.

Answer 1

At a glance my guess is you're mixing classes from the new and old api packages:

ChainMapper (as of 1.1.2) is only implemented for the old API package (org.apache.hadoop.mapred ), and hasn't been ported to the new API package ( org.apache.hadoop.mapreduce ).

As such the second argument to ChainMapper.addMapper ( Class<? extends Mapper<K1,V1,K2,V2>> klass ) is expected to be of type org.apache.hadoop.mapred.Mapper .

As your current mapper NameMapper extends the Mapper class, this leads me to believe that this is the new api Mapper class ( org.apache.hadoop.mapreduce ), as the new API is a class, not an interface

You should be able to fix by amending the signature of your NameMapper class to match that of the old API Mapper interface:

public class NameMapper extends MapReduceBase 
                        implements Mapper<Object, Object, Object, Object> { }

This will also mean your current map method's signature will need to change from

protected void map(Object key, Object value, Context context) { }
// to
public void map(Object key, Object value, OutputCollector collector,
                Reporter reporter) { }

And if you're using the setup and cleanup methods of the new Mapper API, you'll need to replace them too:

public void close() { } // instead of cleanup(Context) {}
public void configure(JobConf conf) { } // instead of setup(Context) {}