Code is giving “Exception in thread ”main“ java.lang.NoClassDefFoundError”

Question

I have added following dependency. When i change spark-streaming_2.12 to spark-streaming_2.11 it gives error this error

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/StreamingContext

Those are my dependencies:

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.12</artifactId>
      <version>2.4.0</version>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.0</version>
    </dependency>

My code:

def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("LogProcessor")
    // No need to create sparkContext as StreamingContext itself create one
    val streamer = new StreamingContext(conf,Seconds(20))
    val lines =  streamer.textFileStream("/home/ubuntu/Desktop/test/")
    println(lines)

  }

Answer 1

I think you may be running this in an environment that does not have the spark-streaming dependency available (and hence, it is not able to find the classes on it).

If that case, you may want to include the dependency in the jar itself. You may try removing the <scope>provided</scope> from the spark-streaming dependency definition.

Edit: To provide more info about why I think this is happening, probably your environment has spark-streaming_2.12 dependency in the classpath, but not the spark-streaming_2.11 one. That is why I think you should include it in an uber jar (removing the provided part). Alternatively, you could also include the dependency in your environment so that your process is able to find it.