Scala代码在spark-shell上有效，但在spark-submit中不起作用

Question

The following is the main scala code

1.val conf=new SparkConf()
2.conf.setMaster("spark://master:7077")
3.conf.setAppName("Commnity Detective")
4.val sc=new SparkContext(conf)
5.val rdd=sc.textFile("hdfs://master:9000/lvcy/test/ungraph/test.txt")
6.val maprdd=rdd.map(line =>{val p=line.split("\\s+");(p(0),p(1))}) union rdd.map( line =>{val p=line.split("\\s+");(p(1),p(0))})
7.val reducerdd=maprdd.reduceByKey((a,b)=>a+"\t"+b)
8.val reduceArray=reducerdd.collect()
9.val reducemap=reduceArray.toMap

Problem statement:

1. copy the code(line:5-9) running on spark-shell, the result is right

2. if put the code to the Eclipse and generate jar packages,then use "spark-submit" to submit the job, there has next error("Main:scala:21" is the top line:9, that is to say the method toMap Error, WHY? ):

    Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at net.lvcy.main.Main$.main(Main.scala:21) at net.lvcy.main.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

Answer 1

It looks like a Scala version mismatch. You should make sure that the Scala version used to generate your jar is the same as the Scala version of your Spark cluster binaries, eg 2.10 .

Answer 2

Prebuild Spark发行版是使用Scala 2.10编译的，因此请确保在Scala 2.10下运行spark集群。