[英]Running Spark example: ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer
I'm new to spark, and yet to write my first spark application and still investigating whether that would be a good fit for our purpose.我是 spark 新手,还没有编写我的第一个 spark 应用程序,并且仍在研究这是否适合我们的目的。 Currently just trying to run the sample example that come with spark that access kafka
目前只是试图运行访问kafka的spark附带的示例示例
I tried to run kafka examples coming out of the box using two ways without success with same error.我尝试使用两种方式运行开箱即用的 kafka 示例,但没有成功并出现相同的错误。
I search existing post but don't quite understand why the out of box don't seem to be working.我搜索现有帖子,但不太明白为什么开箱即用似乎不起作用。
Spark fails with NoClassDefFoundError for org.apache.kafka.common.serialization.StringDeserializer Spark 因 org.apache.kafka.common.serialization.StringDeserializer 出现 NoClassDefFoundError 而失败
Apache Kafka: ...StringDeserializer is not an instance of...Deserializer Apache Kafka:...StringDeserializer 不是...Deserializer 的实例
Why does Spark application fail with "Exception in thread "main" java.lang.NoClassDefFoundError: ...StringDeserializer"? 为什么 Spark 应用程序因“线程“主”java.lang.NoClassDefFoundError: ...StringDeserializer 中的异常而失败?
Clone https://github.com/bitnami/charts.git bitnami/spark
using
registry: docker.io
repository: bitnami/spark
tag: 2.4.5-debian-10-r87
tag: 2.4.5-debian-10-r94
Got success with ./bin/run-example SparkPi 10
But got error with ./bin/run-example streaming.JavaDirectKafkaWordCount myBroker myConsumerGroup myTopic
INFO StreamingExamples: Setting log level to [WARN] for streaming example. To override add a custom log4j.properties to the classpath.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer
at org.apache.spark.examples.streaming.JavaDirectKafkaWordCount.main(JavaDirectKafkaWordCount.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 13 more
Clone https://github.com/apache/spark.git
./build/mvn -DskipTests clean package
[INFO] BUILD SUCCESS
RAN EXAMPLE SUCCESSFULLY
./bin/run-example SparkPi 10
Pi is roughly 3.1424111424111425
RAN KAFKA EXAMPLE WITH ClassNotFoundException
./bin/run-example streaming.JavaDirectKafkaWordCount myBroker myConsumerGroup myTopic
INFO StreamingExamples: Setting log level to [WARN] for streaming example. To override add a custom log4j.properties to the classpath.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer
at org.apache.spark.examples.streaming.JavaDirectKafkaWordCount.main(JavaDirectKafkaWordCount.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:934)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
You need kafka-clients
on the classpath (which should be included as part of spark-sql-kafka-0-10
, which is not provided by Spark by default, so you must download it, and add it to the Spark libs directory)您需要在 classpath 上使用
kafka-clients
(它应该包含在spark-sql-kafka-0-10
一部分中,Spark 默认不提供,因此您必须下载它,并将其添加到 Spark libs 目录中)
Note: Spark "Streaming" is deprecated.注意:Spark“流式传输”已弃用。 You want Structured Streaming
你想要结构化流
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.