![](/img/trans.png)
[英]What is the control flow of Hadoop mapper with MultipleInputs?
[英]Hadoop Mapreduce MultipleInputs cannot load the mapper classes
我在我們的紗線集群上使用了新的MapReduce Api。 我需要從兩個不同的目錄中讀取兩種不同格式的文件。 為此,我決定使用MultipleInputs指定兩個映射器類。 以下是我的工作驅動程序
Job job = new Job(new Configuration(), "Daily Report");
job.setJarByClass(MyDailyJob.class);
MultipleInputs.addInputPath(job, new Path(args[0]), TextInputFormat.class, Record1ParsingMapper.class);
MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class, Record2ParsingMapper.class);
FileOutputFormat.setOutputPath(job, new Path(args[2]));
job.setReducerClass(ReportParsingReducer.class);
job.setNumReduceTasks(10);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);
return (job.waitForCompletion(true) ? 0 : 1);
My Mappers具有以下定義:公共類Record1ParsingMapper擴展了Mapper
運行此作業時,出現以下異常:
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/parser/ParseException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1986)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1951)
at org.apache.hadoop.mapreduce.lib.input.MultipleInputs.getMapperTypeMap(MultipleInputs.java:141)
at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:60)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:498)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:515)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:399)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
at main.java.com.adnear.mr.jobs.MyDailyJob.run(MyDailyJob.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at main.java.com.adnear.mr.jobs.MyDailyJob.main(MyDailyJob.java:226)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.ParseException\
看起來在下面的語句的Configuration類的getClassByName()方法內部失敗。 clazz = Class.forName(name,true,classLoader);
我正在正確指定Mapper類的路徑。 有人可以解釋一下導致此類加載異常的原因嗎?
謝謝,開發人員
Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.ParseException
該錯誤消息明確指出在類路徑中缺少某些類。 此類是運行Job
所需的json-simple-1.1.1.jar
jar文件的一部分,因此請json-simple-1.1.1.jar
jar文件添加到classpath中以成功運行Job。 您可以從此處獲取jar文件。
希望能幫助到你!
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.