Error in running Spark-Scala example in intellij

Question

I Build Scala Project in Intellij and I try to run an Apache Spark example. Here is my code :

package sparkTraining

import org.apache.spark.{SparkConf, SparkContext}

object SimpleApp {
 def main(args: Array[String]) {
   val logFile = "/usr/local/spark/README.md"
   val conf = new SparkConf().setAppName("Simple Application")
   val sc = new SparkContext(conf)
   val logData = sc.textFile(logFile, 2).cache()
   val numAs = logData.filter(line => line.contains("a")).count()
   val numBs = logData.filter(line => line.contains("b")).count()
   println(s"Lines with a: $numAs, Lines with b: $numBs")
   sc.stop()
 }
}

I have installed Scala and SBT plugin on Intellij and here is my build.sbt code:

name := "sparkTraining"

version := "1.0"

scalaVersion := "2.12.0"

and I added all Saprk 2.0.1 jars to the project. But the problem is when I run this piece of code it errors :

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1410)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:77)
at sparkTraining.SimpleApp$.main(testScala.scala:9)
at sparkTraining.SimpleApp.main(testScala.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

I searched for it and I couldn't find any useful answer.

Here is some points :

• I have two user, user1 and hduser , and all my Big Data project like Hadoop, Spark and Scala are under hduser and all paths are set in hduser .bashrc, while Intellij is under root (in /opt folder).
• while executing this code Spark is not running in my machine.

These two points have any affect on executing? How can I solved my problem ? Thanks

Answer 1

I guess it because incompatible of scala version，the official Documents tell that You will need to use a compatible Scala version (2.11.x). official Documents