Error in Spark Streaming code in scala

Question

I am using spark streaming in scala, where I am working with streaming twitter data. I have the following code:

val ssc = new StreamingContext(new SparkConf(), Seconds(5))
val tweets = TwitterUtils.createStream(ssc, None)

val user = tweets.map(x=> x.getText())
val lang = tweets.map(x=> x.getLang())

I am getting the following error:

[error] /home/user/Lab1.1/Twitterstats.scala:103: value getLang is not a member of twitter4j.Status
[error]                 val lang = tweets.map(x=> x.getLang())
[error]                                             ^
[error] one error found

What is wrong with the above code? Could someone please help.

Answer 1

spark-streaming-twitter uses Twitter4j . getLang() is only supported since version 3.0.6 of Twitter4J . If you are using a version 1.5.2 (or lower) of spark-streaming-twitter you won't be able to call getLang() because it uses version 3.0.3 of twitter4j . Since 1.6.0 version 4.0.4 is supported as wel as the getLang() function.

So you could upgrade spark-streaming-twitter to 1.6.0 or higher. Or you could use another 3rd party library to detect the language of your tweets.

(possible duplicate )