Spark 1.3.1 SQL库：scala.reflect.internal.MissingRequirementError：类org.apache.spark.sql.catalyst.ScalaReflection

Question

When trying to run this code:

val conf = new SparkConf()
  .setMaster("local[1]")
  .setAppName("Small")
  .set("spark.executor.memory", "2g")
val sc = new SparkContext(conf)

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val df = sc.parallelize(Array((1,30),(2,10),(3,20),(1,10)(2,30))).toDF("books","readers")
val results = df.join(
df.select($"books" as "r_books", $"readers" as "r_readers"), 
$"readers" === $"r_readers" and $"books" < $"r_books"
)
.groupBy($"books", $"r_books")
.agg($"books", $"r_books", count($"readers"))

Under SBT console started with the following build.sbt:

name := "Small"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.3.1"

Returns error:

scala.reflect.internal.MissingRequirementError: class org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with java.net.URLClassLoader@13a9a4f9 of ...

Any ideas?

Answer 1

you have some spelling errors, and finally the last function count cannot be there you need to import org.apache.spark.sql.functions._

Check this, I corrct the Spelling errors and add the imports

import org.apache.spark.{ SparkContext, SparkConf }
import org.apache.spark.sql.functions._

/**
 * Created by anquegi on 01/06/15.
 */
object QSpark162015 extends App {

  val conf = new SparkConf()
    .setMaster("local[2]")
    .setAppName("QSpark162015")
    .set("spark.executor.memory", "2g")

  val sc = new SparkContext(conf)

  val sqlContext = new org.apache.spark.sql.SQLContext(sc)

  import sqlContext.implicits._

  val df = sc.parallelize(Array((1, 30), (2, 10), (3, 20), (1, 10), (2, 30))).toDF("books", "readers")
  val results = df.join(
    df.select($"books" as "r_books", $"readers" as "r_readers"),
    $"readers" === $"r_readers" and $"books" < $"r_books"
  )
    .groupBy($"books", $"r_books")
    .agg($"books", $"r_books", count($"readers"))

  results.foreach(println _)

  sc.stop()

}