Spark 2.1.0与Kafka 0.9.0的连接

Question

I am using Kafka 0.9.0 and Spark 2.1.0 My Spark Submit is as follows:

./spark-submit --jars /home/cnbo/jars/spark-sql-kafka-0-10_2.11-2.1.0.cloudera1.jar --class ClickStream /home/cnbo/jars/sparkstreamingfi_2.11-0.1.jar

I am getting following error:

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

I know there is version conflict for 0.9 to 0.10. But I want to run My application on Kafka 0.9.0. What needs to be done now? what external jar I should use instead of:

spark-sql-kafka-0-10_2.11-2.1.0.cloudera1.jar

Thanks in Advance!!

Answer 1

In order to use Spark along with Kafka, you need to add following JAR too in spark.driver.extraClassPath & spark.executor.extraClassPath :

/home/cnbo/jars/kafka-clients-0.9.0.0.jar

Since spark-sql-kafka-0-10_2.11-2.1.0.cloudera1.jar does not contain KafkaConsumer , we need to add above-mentioned JAR file too. So, the final spark-submit command will be like this:

./spark-submit --jars /home/cnbo/jars/spark-sql-kafka-0-10_2.11-2.1.0.cloudera1.jar:/home/cnbo/jars/kafka-clients-0.9.0.0.jar --class ClickStream /home/cnbo/jars/sparkstreamingfi_2.11-0.1.jar

I hope it helps!

Answer 2

If you read the Spark Kafka page, you'd have seen

0.8 integration is compatible with later 0.9 and 0.10 brokers, but the 0.10 integration is not compatible with earlier brokers

Therefore, you need this library

spark-streaming-kafka-0-8_2.11

And if you want structured streaming, you need to actually upgrade Kafka to support the new consumer API