[英]ClassNotFoundException for Kafka-Spark cluster
I'm trying to run some .jar file made in scala-Spark 2.0.2. 我正在尝试运行一些在scala-Spark 2.0.2中制作的.jar文件。 in my Spark-Kafka cluster.
在我的Spark-Kafka集群中。 The code's here:
代码在这里:
import java.util.HashMap
import org.apache.kafka.clients.producer.{KafkaProducer,ProducerConfig}
import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
object sparkKafka{
def main(args: Array[String]): Unit = {
if(args.length < 4){
System.err.println("Usage: sparkKafka <zkQuorum><group> <topics> <numThreads>")
System.exit(1)
}
val Array(zkQuorum, group, topics, numThreads) = args
val sparkConf = new SparkConf()
.setAppName("sparkKafka")
val ssc = new StreamingContext(sparkConf, Seconds(2))
ssc.checkpoint("E:/temp/")
val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2)
val words = lines.flatMap(_.split(" "))
val wordsCounts = words.map(x => (x, 1L))
.reduceByKeyAndWindow(_ + _, _ - _, Minutes(10), Seconds(2), 2)
wordsCounts.print()
ssc.start()
ssc.awaitTermination()
} }
} }
I built .jar file named: kafka-spark.jar and scp-it to my node in spark2 folder so it could read it. 我建立了一个名为.kafka-spark.jar的 .jar文件,并把它scp-it到spark2文件夹中的节点,以便它可以读取它。 Afterwards I went to start the script with:
之后,我使用以下命令启动脚本:
bin/spark-submit --class "sparkKafka" --master local[4] kafka-spark.jar hdp2.local:2181 group1 Topic-example 1 -verbose
The error I'm getting is like it said in the head of topic, or ClassNotFoundException: sparkKafka 我收到的错误就像在主题标题或ClassNotFoundException:sparkKafka中说的那样
[root@hdp2 spark2]# bin/spark-submit --class "sparkKafka" --master local[4] kafka-spark.jar hdp2.local:2181 group1 Topic-example 1 -verbose
java.lang.ClassNotFoundException: sparkKafka
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Where am I making mistake? 我在哪里犯错? Also I tried with full path to my jar file, but eihter I get that .jar not found or this error above.
我也尝试使用jar文件的完整路径,但是后来我发现找不到.jar或上面的错误。 Also I tried without -v but I think it doesn't make any difference.
我也尝试了不带-v的情况,但我认为这没有任何区别。 It would be great if someone knows where is a problem.
如果有人知道问题出在哪里,那就太好了。 Thank you!
谢谢!
Did you try calling it like this 您是否尝试过这样称呼它
bin/spark-submit --class "sparkKafka" --master local[4] kafka-spark.jar hdp2.local:2181 group1 Topic-example 1 -verbose
If that does not work you might want to extract content of kafka-spark.jar and check if it actually contains sparkKafka class. 如果这样不起作用,则可能要提取kafka-spark.jar的内容并检查它是否实际上包含sparkKafka类。
Hava您将包添加到sparkKafka.scala吗?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.