[英]deeplearning4j on yarn cluster fails
我想使用master = yarn-cluster将我的jar提交到集群上,但是会出错。
java.lang.NoClassDefFoundError: org/nd4j/Nd4jRegistrator
at Main.main(Main.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
Caused by: java.lang.ClassNotFoundException: org.nd4j.Nd4jRegistrator
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
我使用krio序列化器这样设置sparkConf,因为在没有它的情况下运行时会出现异常
SparkConf sparkConf = new SparkConf().setMaster(master).setAppName("DL4J Spark Example");
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
sparkConf.set("spark.kryo.registrator", "org.nd4j.Nd4jRegistrator");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
pom文件中的依赖项:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>0.8.0</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>dl4j-spark_2.10</artifactId>
<version>0.8.0_spark_1</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native-platform</artifactId>
<version>0.8.0</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-kryo_2.10</artifactId>
<version>0.8.0</version>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
它找不到Nd4jRegistrator类,但是为什么呢? 当我在本地运行时,它可以正常工作。
我猜您没有正确设置uberjar? 您需要确保捆绑所有内容。 您的pom总体看起来还不错。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.