[英]Exception while starting java spark-streaming application
该应用程序引发java.lang.NoSuchMethodException
堆栈跟踪
DAGScheduler: Failed to run runJob at ReceiverTracker.scala:275
Exception in thread "Thread-33" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 77, 172.20.7.60): java.lang.NoSuchMethodException: org.apache.spark.examples.streaming.KafkaKeyDecoder.<init>(kafka.utils.VerifiableProperties)
java.lang.Class.getConstructor0(Class.java:2810)
java.lang.Class.getConstructor(Class.java:1718)
org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:106)
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
似乎此链接已在spark 1.1.0中修复了问题
火花:1.1.0
卡夫卡:0.8.1.1
就我而言,如评论中所述,通过消除库冲突,我能够正确地使用kafka中的数据并将其存储到cassandra中,然后将该作业部署到Datastax Analytics解决方案中。 我发现与开放源代码不同的是,streaming_kafka jar和所有scala库已经包含在执行程序类路径中。
所以我建议如下:
我假设您正在构建尝试部署的uber jar。
您缺少包含该方法的Kafka jar。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.