NoClassDefFound 异常 Spark Streaming

Question

I am trying to run the sample JavaStatefulNetworkWordCount Algorithm provided by Apache Spark examples but am experiencing a problem when I try to run the program using spark submit, I get the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/StateSpec
    at JavaStatefulNetworkWordCount.main(JavaStatefulNetworkWordCount.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.StateSpec
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

I have imported the StateSpec classes and the code is the same as the one provided over here:
https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java

I would appreciate any help in understanding why this problem arises and how I can fix it.

Answer 1

You should probably add the actual submit command as well.

I highly suspect that you are either missing the --class parameter at all when submitting the spark jar or you did not use the correct qualified class name in it.

Answer 2

I tried using the following command:

spark-submit --class JavaStatefulNetworkWordCount --master local[2] target/SparkStreaming.jar localhost 2222