[英]Spark NoSuchMethodError on SQLContext.sql (Spark 1.6.0 on Cloudera 5.8.0)
[英]Twitter receiver not running in spark 1.6.0
出於某種原因,如果我包含一個 twitter 接收器並啟動流上下文,我會得到以下異常,不知道為什么有人可以讓我知道是否有人已經遇到過這個問題,或者我做錯了什么?
java.lang.ArithmeticException: / by zero
at org.apache.spark.streaming.Duration.isMultipleOf(Duration.scala:59)
at org.apache.spark.streaming.dstream.DStream.isTimeValid(DStream.scala:324)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344)
at scala.Option.orElse(Option.scala:257)
at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:341)
at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:47)
at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:115)
at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:114)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:114)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:248)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:246)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:246)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:181)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:87)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:86)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
2016-03-28 10:08:00,112 ERROR [DefaultQuartzScheduler_Worker-8] org.quartz.core.JobRunShell
Job sample_TwitterListener.SocialMedia_sample threw an unhandled Exception:
終於找到了問題所在,在代碼中我正在初始化流上的兩個窗口,如下所示
JavaReceiverInputDStream<Status> inputStream = TwitterUtils.createStream(jssc, getAuth());
JavaDStream<String> batchedInput = rawInput.window(new Duration(windowInterval), new Duration(slideInterval));
processStreamData(batchedInput);
private void processStreamData(JavaDStream<String> _input) {
JavaDStream<String> input = _input.window(new Duration(windowInterval), new Duration(slideInterval));
}
因此,第一個窗口流用於獲得正確的滑動間隔,但不知何故第二個窗口獲得了 0 ms 的滑動間隔,這導致 / 零異常。 刪除第二個窗口操作后,我能夠收到推特數據。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.