[英]Spark Streaming - java.lang.NoSuchMethodError Error
I am trying to access the streaming tweets from Spark Streaming. 我正在尝试从Spark Streaming访问流式推文。
This is the software configuration. 这是软件配置。
Ubuntu 14.04.2 LTS
Ubuntu 14.04.2 LTS
scala -version scala -version
Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
Scala代码运行器版本2.11.7 - 版权所有2002-2013,LAMP / EPFL
spark-submit --version spark-submit --version
Spark version 1.6.0
Spark版本1.6.0
Following is the code. 以下是代码。
object PrintTweets
{
def main(args: Array[String]) {
// Configure Twitter credentials using twitter.txt
setupTwitter()
// Set up a Spark streaming context named "PrintTweets" that runs locally using
// all CPU cores and one-second batches of data
val ssc = new StreamingContext("local[*]", "PrintTweets", Seconds(1))
// Get rid of log spam (should be called after the context is set up)
setupLogging()
// Create a DStream from Twitter using our streaming context
val tweets = TwitterUtils.createStream(ssc, None)
// Now extract the text of each status update into RDD's using map()
val statuses = tweets.map(status => status.getText())
// Print out the first ten
statuses.print()
// Kick it all off
ssc.start()
ssc.awaitTermination()
}
}
Utilities.scala Utilities.scala
object Utilities {
/** Makes sure only ERROR messages get logged to avoid log spam. */
def setupLogging() = {
import org.apache.log4j.{Level, Logger}
val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)
}
/** Configures Twitter service credentials using twiter.txt in the main workspace directory */
def setupTwitter() = {
import scala.io.Source
for (line <- Source.fromFile("./data/twitter.txt").getLines) {
val fields = line.split(" ")
if (fields.length == 2) {
System.setProperty("twitter4j.oauth." + fields(0), fields(1))
}
}
} }
} }
Issues: 问题:
Since it needs the twitter4j library, i have added 由于它需要twitter4j库,我已经添加了
twitter4j-core-4.0.4, twitter4j-stream-4.0.4 in eclipse build path as external jars. twitter4j-core-4.0.4,twitter4j-stream-4.0.4在eclipse构建路径中作为外部jar。
Then i ran the program, it didnt throw any error. 然后我运行程序,它没有抛出任何错误。 But the tweets not appearing in console.
但这些推文没有出现在控制台中。 It were empty.
它是空的。
So i see some forums and downgraded twitter4j to 3.0.3. 所以我看到一些论坛并将twitter4j降级为3.0.3。 Also in Eclipse i chosen Scala 2.10 Library container in Build Path window.
同样在Eclipse中,我在Build Path窗口中选择了Scala 2.10 Library容器。
After that i got java.lang.NoSuchMethodError run-time error. 之后我得到java.lang.NoSuchMethodError运行时错误。
16/05/14 11:46:01 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StreamListener;)V
at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Please help me to resolve this. 请帮我解决这个问题。 Initially i have installed spark by built using Scala 2.11.
最初我使用Scala 2.11构建了spark。 Is that the problem.
那是问题所在。 Do i need uninstall everything and re-install Scala 2.10, then Spark pre-compiled package.
我是否需要卸载所有内容并重新安装Scala 2.10,然后是Spark预编译包。
Or apart from Scala 2.11, do i need to have Scala 2.10 in my system? 或者除了Scala 2.11之外,我需要在我的系统中安装Scala 2.10吗?
The above exception seems to be caused by the incompatibility of spark version 1.6.0
and twitter4j 3.0.3 version
. 上述异常似乎是由
spark version 1.6.0
和twitter4j 3.0.3 version
的不兼容引起的。
twitter4j.TwitterStream
which is being passed in the onStart
method of org.apache.spark.streaming.twitter.TwitterReceiver
, has method addListener which takes instance of twitter4j.StreamListener
. 在
org.apache.spark.streaming.twitter.TwitterReceiver
的onStart
方法中传递的twitter4j.TwitterStream
有方法addListener,它接受twitter4j.StreamListener
实例。
twitter4j 3.0.3 version
has no method twitter4j.TwitterStream.addListener(StreamListener)
, instead it has few other addListener
methods, which take the subclass of StreamListener
. twitter4j 3.0.3 version
没有方法twitter4j.TwitterStream.addListener(StreamListener)
,而是几乎没有其他addListener
方法,它们采用StreamListener
的子类。
twitter4j 4.0.4 version
has the desired method, so that's why no error comes with this library. twitter4j 4.0.4 version
有所需的方法,这就是为什么这个库没有错误的原因。 So changing to twitter4j 3.0.3 version
will not solve the problem. 所以更改为
twitter4j 3.0.3 version
将无法解决问题。
Problem is somewhere else. 问题出在其他地方。
In my case. 就我而言。 I had spark java project.
我有火花java项目。 I cleaned pom file and start adding in order.
我清理了pom文件并开始按顺序添加。 First resolved spark related errors, then spark launcher, next on ward based on bigger library.
首先解决火花相关的错误,然后火花发射器,在病房的基础上更大的库。 Note I was using cdh6.2.0 environment
注意我使用的是cdh6.2.0环境
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.