Can use spark streaming 1.5.1 with kafka 0.10.0?

Question

Can I use spark streaming 1.5.1 with kafka 0.10.0? The site spark.apache.org recommended that spark streaming should work with kafka 0.8.2;

I just wanna know what if I use kafka 0.10.0 with spark 1.5.1 ？

Can someone please help.

Answer 1

I think it's possible but there are some limitations. For instance, Apache spark 1.5.x does not support features like streaming in a kerberized environment.

Answer 2

This API spark-streaming-kafka-0-10 is still in experimental condition. You can use this. But you might get some surprises as it is yet to stable. Please follow the below link:

https://github.com/jerryshao/spark-streaming-kafka-0-10-connector

This is a Kafka 0.10 connector for Spark 1.x Streaming. Hope it will work.