Why does the Scala compiler fail with missing parameter type for filter with JavaSparkContext?

Question

I am trying to add filter as shown in the examples to my program:

val logFile = "/tmp/master.txt"
val sc = new JavaSparkContext("local[4]", "Twitter Analyzer", "/home/welcome/Downloads/spark-1.1.0/",Array("target/scala-2.10/Simple-assembly-0.1.0.jar"))
val twitterFeed = sc.textFile(logFile).cache()

while (iterator.hasNext) {
  val value = iterator.next()
  val numAs = twitterFeed.filter(line => line.contains(value))
  numAs.saveAsTextFile("/tmp/output/positive/" + value)
}

I get the compilation error as follows:

[info] Compiling 1 Scala source to /home/siva/file1/target/scala-2.10/classes...
[error] /home/siva/file1/src/main/scala/com/chimpler/example/twitter/Tweet.scala:27: missing parameter type
[error]     val numAs = twitterFeed.filter(line => line.contains(value))
[error]                                    ^
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 5 s, completed 19 Sep, 2014 1:31:26 PM

any ideas?

Answer 1

As @groverboy advised in the comment you should really be using org.apache.spark.SparkContext instead. Spark Programming Guide's Initializing Spark is also clear on this.

import org.apache.spark._

val conf = new SparkConf()
  .setMaster("local[4]")
  .setAppName("Twitter Analyzer")
  .setSparkHome("/home/welcome/Downloads/spark-1.1.0/")
  .setJars(Seq("target/scala-2.10/Simple-assembly-0.1.0.jar"))
val sc = new SparkContext(conf)

The reason for this is the type inference in Scala that needs type context to infer the type of line parameter.

val numAs = twitterFeed.filter(line => line.contains(value))

It's clearly of String type, but using the Java version of SparkContext - JavaSparkContext - you simply lose the type information.

Provided you use SparkContext the above line could be further simplified to:

val numAs = twitterFeed.filter(_.contains(value))

or even:

twitterFeed.filter(_ contains value)

All the goodies just SparkContext away.

Answer 2

val numAs =  twitterFeed.filter((i: String) => i.contains(value))

解决了这个问题。