Spark Structured Streaming getting messages for last Kafka partition

Question

I am using Spark Structured Streaming to read from Kafka topic.

Without any partition, Spark Structired Streaming consumer can read data.

But when I added partitions to topic, the client is showing messages from last partition only. Ie if there are 4 partitions in topic and I.am pushing numbers like 1,2,3,4 in topic,then client printing only 4 not other values.

I am using latest samples and binaries from Spark Structured Streaming website.

    DataFrame<Row> df = spark
 .readStream()
 .format("kafka") 
.option("kafka.bootstrap.servers", "host1:port1,host2:port2") 
.option("subscribe", "topic1") 
.load()

Am I missing anything?

Answer 1

Issue is resolved by changing kafka-clients-0.10.1.1.jar to kafka-clients-0.10.0.1.jar.

Found reference here Spark Structured Stream get messages from only one partition of Kafka