致命[主要] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：启动MRAppMaster java.lang.NoClassDefFoundError时出错
[英]FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoClassDefFoundError
问题描述
I am new to hadoop. 
我是hadoop的新手。
I am trying to setup Giraph to run on hadoop-2.6.5 with yarn. 我正在尝试将Giraph设置为在带有纱线的hadoop-2.6.5上运行。
When I submit the Giraph job the job gets submitted successfully but fails and I get below log in container syslog: 当我提交Giraph作业时，该作业被成功提交但失败，并且在容器syslog中获得以下日志：
2018-01-30 12:09:01,190 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1517293264136_0002_000002 2018-01-30 12:09:01,437 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-01-30 12:09:01,471 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2018-01-30 12:09:01,471 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 2 cluster_timestamp: 1517293264136 } attemptId: 2 } keyId: -1485907628) 2018-01-30 12:09:01,583 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
Diagnosis in logs shows following log: 日志中的诊断显示以下日志：
Application application_1517293264136_0002 failed 2 times due to AM Container for appattempt_1517293264136_0002_000002 exited with exitCode: 1 For more detailed output, check application tracking page: http://172.16.0.218:8088/proxy/application_1517293264136_0002/Then , click on links to logs of each attempt.
The class it is failing at is io/netty/buffer/ByteBufAllocator which is in netty-all jar: https://mvnrepository.com/artifact/io.netty/netty-all 它失败的类是io/netty/buffer/ByteBufAllocator ，它位于netty-all jar中： https : io/netty/buffer/ByteBufAllocator
From other questions I have tried adding the jar in HADOOP_CLASSPATH. 从其他问题，我尝试将罐子添加到HADOOP_CLASSPATH中。
Yogin-Patel:hadoop yoginpatel$ echo $HADOOP_CLASSPATH
/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar
Yogin-Patel:hadoop yoginpatel$
It shows up in hadoop classpath as well. 它也显示在hadoop classpath中。
Yogin-Patel:hadoop yoginpatel$ hadoop classpath
/Users/yoginpatel/hadoop/etc/hadoop:/Users/yoginpatel/hadoop/share/hadoop/common/lib/*:/Users/yoginpatel/hadoop/share/hadoop/common/*:/Users/yoginpatel/hadoop/share/hadoop/hdfs:/Users/yoginpatel/hadoop/share/hadoop/hdfs/lib/*:/Users/yoginpatel/hadoop/share/hadoop/hdfs/*:/Users/yoginpatel/hadoop/share/hadoop/yarn/lib/*:/Users/yoginpatel/hadoop/share/hadoop/yarn/*:/Users/yoginpatel/hadoop/share/hadoop/mapreduce/lib/*:/Users/yoginpatel/hadoop/share/hadoop/mapreduce/*:/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar:/contrib/capacity-scheduler/*.jar
Yogin-Patel:hadoop yoginpatel$
I am trying to setup in development environment. 我正在尝试在开发环境中进行设置。 This is single node setup. 这是单节点设置。
I have even tried 我什至尝试过
job.addFileToClassPath(new Path("/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar"));
None of the approaches helped. 这些方法都无济于事。 How do I make hadoop node get the necessary jar accessed? 如何使hadoop节点访问必要的jar？
This is a GiraphJob submit code which would submit map reduce job to the cluster: 这是一个GiraphJob提交代码，该代码将地图减少作业提交到集群：
@Test
public void testPageRank() throws IOException, ClassNotFoundException, InterruptedException {
GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
giraphConf.setWorkerConfiguration(1,1,100);
GiraphConstants.SPLIT_MASTER_WORKER.set(giraphConf, false);
giraphConf.setVertexInputFormatClass(JsonLongDoubleFloatDoubleVertexInputFormat.class);
GiraphFileInputFormat.setVertexInputPath(giraphConf,
new Path("/input/tiny-graph.txt"));
giraphConf.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);
giraphConf.setComputationClass(PageRankComputation.class);
GiraphJob giraphJob = new GiraphJob(giraphConf, "page-rank");
giraphJob.getInternalJob().addFileToClassPath(new Path("/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar"));
FileOutputFormat.setOutputPath(giraphJob.getInternalJob(),
new Path("/output/page-rank2"));
giraphJob.run(true);
}
private Configuration getConf() {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://localhost:9000");
conf.set("yarn.resourcemanager.address", "localhost:8032");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
return conf;
}
1 个解决方案
解决方案1
0 2018-01-31 14:55:01
我通过将带有依赖项的giraph的jar放在hadoop lib路径中来使其工作：
cp giraph-1.3.0-SNAPSHOT-for-hadoop-2.6.5-jar-with-dependencies.jar ~/hadoop/share/hadoop/mapreduce/lib/
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.