编译Spark类时，“ Scala Eclipse插件”中出现错误

Question

I am using CDH5.1.0 to do some simple Spark programming. Also, I have Eclipse Juno (comes with the VM) and installed Scala IDE plugin 2.10.0. I am getting the following error in the IDE:

Bad symbolic reference. A signature in SparkContext.class refers to term io in package org.apache.hadoop which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling SparkContext.class. SimpleApp.scala /MyScalaProject/src/com/test/spark1 line 10 Scala Problem

Code:

package com.test.spark1
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/home/Desktop/scala/sparktest.txt" // Should be some file on your system
    val conf = new org.apache.spark.SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s").format(numAs, numBs)
  }
}

I get the same error at line# 10 (var conf - new org.apache.spark.SparkCon...) and also line# 15 (println...).

My project build path has /usr/lib/spark/assembly/lib/spark-assembly-1.0.0-cdh5.1.0-hadoop2.3.0-cdh5.1.0.jar and I checked all necessary classes for this simple scala program are there.

Answer 1

The compilation error went away once I added the following jar in the build path:

hadoop-common-2.3.0-cdh5.1.0.jar

so there was some internal dependency that was missing causing this error.