映射減少實例化異常

Question

嗨，我有以下map-reduce代碼，通過這些代碼我試圖解析XML文件並在輸出中創建CSV。

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class XMLParseMR {

   public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
      private final static IntWritable one = new IntWritable(1);
      public static String outputFile = null;
      private Text word = new Text();
      private JAXBC jax = new JAXBC();

      public void map(LongWritable key, Text value, Context context) throws 
          IOException, InterruptedException {

        String document = value.toString();
        System.out.println("XML : "+ document);
        try {
          ConnectHome ch = jax.convertJAXB(document);
      jax.convertCSV(ch, outputFile);
    } 
        catch (JAXBException e) {
          // TODO Auto-generated catch block
      e.printStackTrace();
    }
         } 
      }

 public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = new Job(conf, "wordcount");
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    job.setMapperClass(Map.class);
    conf.set("xmlinput.start", "<ConnectHome>");          
    conf.set("xmlinput.end", "</ConnectHome>");     
    job.setInputFormatClass(XMLInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    Map.outputFile = args[1];
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));


    job.waitForCompletion(true);
   }
  }

我還有一個名為Connect_Home的類，其中我對數據進行解析，這些數據使用JAXB提取數據。 但是，當我運行代碼時，出現以下錯誤：

WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. 
        Applications should implement Tool for the same.
    WARN mapred.JobClient: No job jar file set.  User classes may not be found. See      
        JobConf(Class) or JobConf#setJar(String).
    INFO input.FileInputFormat: Total input paths to process : 1
    INFO util.NativeCodeLoader: Loaded the native-hadoop library
    WARN snappy.LoadSnappy: Snappy native library not loaded
    INFO mapred.JobClient: Running job: job_201303121556_0011
    mapred.JobClient:  map 0% reduce 0%
    INFO mapred.JobClient: Task Id : attempt_201303121556_0011_m_000000_0, Status : 
         FAILED
    java.lang.RuntimeException: java.lang.ClassNotFoundException: XMLParseMR$Map
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1004)
        org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:217)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at 

       org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
            at org.apache.hadoop.mapred.Child.main(Child.java:260)
    Caused by: java.lang.ClassNotFoundException: XMLParseMR$Map
            at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
            at sun.misc.Launcher$AppClassLoader.loadCl
    INFO mapred.JobClient: Task Id : attempt_201303121556_0011_m_000000_1, Status :  
       FAILED
        java.lang.RuntimeException: java.lang.ClassNotFoundException: 
       XMLParseMR$Map
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1004)
            at 
       org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:217)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at 
      org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
            at org.apache.hadoop.mapred.Child.main(Child.java:260)
    Caused by: java.lang.ClassNotFoundException: XMLParseMR$Map
            at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.misc.Launcher$AppClassLoader.loadCl
    INFO mapred.JobClient: Task Id : attempt_201303121556_0011_m_000000_2, Status : 
       FAILED
    java.lang.RuntimeException: java.lang.ClassNotFoundException: XMLParseMR$Map
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1004)
            at 
      org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:217)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at 

       org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
            at org.apache.hadoop.mapred.Child.main(Child.java:260)
    Caused by: java.lang.ClassNotFoundException: XMLParseMR$Map
            at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
            at sun.misc.Launcher$AppClassLoader.loadCl
    INFO mapred.JobClient: Job complete: job_201303121556_0011
    INFO mapred.JobClient: Counters: 7INFO mapred.JobClient:   Job Counters
    INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=20097
    INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving 
      slots (ms)=0
    INFO mapred.JobClient:     Total time spent by all maps waiting after reserving 
      slots (ms)=0
    INFO mapred.JobClient:     Launched map tasks=4
    INFO mapred.JobClient:     Data-local map tasks=4
    INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
    INFO mapred.JobClient:     Failed map tasks=1

Answer 1

錯誤信息：

WARN mapred.JobClient: No job jar file set.  User classes may not be found. See      
    JobConf(Class) or JobConf#setJar(String).

告訴您作業未正確設置。 不用按名稱設置JAR，您可以讓Hadoop通過在設置作業時調用setJarByClass()自行確定它：

Job job = new Job(conf, "wordcount");
job.setJarByClass(XMLParseMR.class);

它將根據您的類名稱設置Job的 JAR。 之后，您的作業應正確運行，並且上述錯誤消息將消失。