Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.longArrayOps([J)[J

Question

I have a simple Scala object that creates an RDD and then collects and prints out all the elements.

I've created a Maven project on Eclipse and added Scala library 2.12.3 To pom.xml I have added spark 2.4.3 dependency as below :

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>2.4.3</version>
</dependency>

Finally, I've created a JAR and am then trying to execute spark-submit but this is failing with

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.longArrayOps([J)[J
    at org.spark.learning.Demo$.main(Demo.scala:14)
    at org.spark.learning.Demo.main(Demo.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The culprit seems to be r1.collect.foreach(println) in my scala code where r1 is the rdd created from range(1,50)

And yes, I have Scala 2.12.3 and Spark 2.4.3 on Eclipse as well as my terminal so version incompatibility doesn't seem to be the issue here.

Could someone please help ?

Answer 1

this is clear version issue. nothing else even though you are claiming to use 2.12.x seems like its pointing to old version of scala try to clean and build. verify dependencies in maven or sbt which ever you are using.

Also do File -> Project Structure -> Global libraries -> Remove SDK -> Rebuild

if you are using intellij all external libraries under class path will be listed like in below picture... under external libraries section.

one way to find the decrepencies is using classloader...

val  urls = urlsinclasspath(getClass.getClassLoader).foreach(println)


def urlsinclasspath(cl: ClassLoader): Array[java.net.URL] = cl match {
    case null => Array()
    case u: java.net.URLClassLoader => u.getURLs() ++ urlsinclasspath(cl.getParent)
    case _ => urlsinclasspath(cl.getParent)
  }

using this you can print all the jars which are in the classpath of the project which you are running from intellij or your driver program using cluster.