Spark和Kafka问题-线程“ main”中的异常java.lang.NoClassDefFoundError：org.apache.spark.streaming.kafka010.LocationStrategies

Question

I will start by mentioning that I have tried all the suggestion in similar topics and nothing worked for me, so please, this is not a duplicate question .

The issue that I am having is as follows -

I am trying to run a sample java application on spark, using spark streaming and kafka. I have added all the required dependencies:

  <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.12</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>2.3.0</version>
        </dependency>
    </dependencies>

After deploying the Jar on a server where I would like to run my application (I have already set up the environment with spark, kafka, and created the relevant topic) I am trying to spark-submit it and getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.streaming.kafka010.LocationStrategies
        at JavaWordCount.main(JavaWordCount.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka010.LocationStrategies
        at java.net.URLClassLoader.findClass(URLClassLoader.java:609)
        at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:924)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:869)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:852)

It seems that the workers cannot identify the dependencies as part of the environment. I did some research online, and many suggest to create an assembly JAR with maven-shade-plugin. So I also tried to maven-package the jar this way, but still no success.

For reference, here is where the app is failing:

// Configure Spark to connect to Kafka running on local machine
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, 
                "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, 
                "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG,"group1");
kafkaParams.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"latest");
kafkaParams.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,true);

//Configure Spark to listen messages in topic test
Collection<String> topics = Arrays.asList("wordCount");

SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("SparkKafkaWordCount");

//Read messages in batch of 30 seconds
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(30));

// Start reading messages from Kafka and get DStream
final JavaInputDStream<ConsumerRecord<String, String>> stream =
        KafkaUtils.createDirectStream(jssc, LocationStrategies.PreferConsistent(), 
                                      ConsumerStrategies.<String,String>Subscribe(topics,kafkaParams));

In the last line above, the class LocationStrategies is not recognized, even though I have added the right dependency to the pom.xml

Any ideas how to fix this issue?

Answer 1

当我在spark-spark-submit命令的--jar下包含以下jar时，没有出现此错误：

• sql-kafka-0-10_2.11-2.2.1.jar
• kafka-clients-0.10.1.0.jar
• spark-streaming-kafka-0-10_2.11-2.2.1.jar
• spark-streaming-kafka-0-10-assembly_2.11-2.2.1.jar
• spark-streaming-kafka_2.11-1.6.3.jar

Answer 2

Even I faced the same issue, didn't get much help from googling but what I've understood after reading many threads, the dependencies which are mentioned in pom.xml file has scope as "provided" which means we need to specify the dependent jar files during the time of execution. Also all the examples inside Apache Spark package was compiled as a single jar file and we need to specify the class path to execute the required module. Download the necessary jar files which you've mentioned in pom.xml and execute like this

spark-submit --jars kafka-clients-1.1.0.jar,spark-streaming_2.11-2.3.0.jar,spark-streaming-kafka-0-10_2.11-2.3.0.jar --class org.apache.spark.examples.streaming.JavaDirectKafkaWordCount target/original-spark-examples_2.11-2.4.0-SNAPSHOT.jar <brokerip:port> <topic>

But before that you need to re-write the consumer properties in the java file or else you will get error saying configuration is missing

kafkaParams.put("bootstrap.servers", brokers);
kafkaParams.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("group.id", "<group_id>");

Answer 3

add this to your pom.xml

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>

            <configuration>
                <archive>
                    <manifest>
                        <mainClass>your main class </mainClass>
                    </manifest>
                </archive>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
            </configuration>
        </plugin>

and then build the jar with

mvn clean compile assembly:single

You should get two jars in the target directory, one without dependencies and one with dependencies (your-jar-1.0-jar-with-dependencies.jar)