Spark 2.1.0错误-kafka.cluster.BrokerEndPoint无法转换为kafka.cluster.Broker

Question

I have seen several other questions on SO about this that indicate this is a dependency/version issue, eg kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.Broker

However, I can't tell what is wrong with my dependencies. I'm using Spark 2.1.0, and

org.apache.spark.spark-core_2.11
org.apache.spark.spark-sql_2.11
org.apache.spark.spark-streaming-kafka-0-8_2.11
org.apache.spark.spark-streaming_2.11

as dependencies in my job jar I am trying to run via spark-submit . I know that the broker version for the Kafka server is 0.8, and these depedences are all set to 2.1.0. All of these dependencies seem compatible, so I am not sure where my error is.

Edit: I have discovered that if I change the spark-streaming-kafka-0-8_2.11 to be a provided dependency and pass it when I call spark-submit , it works. I am not sure why this is and why I can't just bundle it in the main jar with everything else.

Answer 1

检查以确保在运行时类路径中的某处没有任何可能被加载的Kafka jar。