Why does a Spark application fail with “Exception in thread ”main“ java.lang.NoClassDefFoundError” in Eclipse?

Question

I'm trying to run a Spark application in Eclipse with JRE 1.8.0 and am facing the following exception. Why?

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/04/01 17:05:30 INFO SparkContext: Running Spark version 1.4.1
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
   at org.apache.spark.util.TimeStampedWeakValueHashMap.<init>(TimeStampedWeakValueHashMap.scala:42)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:277)
   at com.pooja.spark.training.sparkexamples.wordcount$.main(wordcount.scala:10)
   at com.pooja.spark.training.sparkexamples.wordcount.main(wordcount.scala)
Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
   at java.net.URLClassLoader.findClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
   at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
   ... 4 more

Answer 1

"Spark version 1.4.1" ?! That's old. Unless you know what you're doing, please upgrade to the latest in 1.x release chain or 2.3.0 if possible.

---

Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class

The runtime exception is because you don't have Scala libraries available in your runtime environment (in the JVM's CLASSPATH). You have to tell Eclipse to use Scala as a dependency that has to be included in the JVM when running the Spark application.

I recommend using Eclipse with tools like sbt or Apache Maven to manage dependencies.

While we're at it, I also recommend trying out the Spark examples in Apache Spark Examples .