Kafka - Spark Streaming - Reading Data from only 1 partition

Question

I have a standalone spark cluster that is reading data from a kafka queue. The kafka queue had 5 partitions, spark is only processing data from one of the partitions. I'm using the following:

Here are my maven dependencies:

    <dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <version>2.0.2</version>
    </dependency>   
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.0.2</version>
    </dependency>
    <dependency>
        <groupId>kafka-custom</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.1.1</version>
    </dependency>   

My kafka producer is a simple producer that is just putting 100 messages on the queue:

    public void generateMessages() {

    // Define the properties for the Kafka Connection
    Properties props = new Properties();
    props.put("bootstrap.servers", kafkaBrokerServer); // kafka server
    props.put("acks", "all");
    props.put("retries", 0);
    props.put("batch.size", 16384);
    props.put("linger.ms", 1);
    props.put("buffer.memory", 33554432);
    props.put("key.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");

    // Create a KafkaProducer using the Kafka Connection properties
    KafkaProducer<String, String> producer = new KafkaProducer<String, String>(
            props);
    for (int i = 0; i < 100; i++) {
        ProducerRecord<String, String> record = new ProducerRecord<>(kafkaTopic, "value-" + i);
        producer.send(record);
    }
    producer.close();

}

Here is the main code in my spark streaming job:

    public void processKafka() throws InterruptedException {
    LOG.info("************ SparkStreamingKafka.processKafka start");

   // Create the spark application
    SparkConf sparkConf = new SparkConf();
    sparkConf.set("spark.executor.cores", "5");

    //To express any Spark Streaming computation, a StreamingContext object needs to be created. 
    //This object serves as the main entry point for all Spark Streaming functionality.
    //This creates the spark streaming context with a 'numSeconds' second batch size
    jssc = new JavaStreamingContext(sparkConf, Durations.seconds(sparkBatchInterval));


    //List of parameters
    Map<String, Object> kafkaParams = new HashMap<>();
    kafkaParams.put("bootstrap.servers", this.getBrokerList());
    kafkaParams.put("client.id", "SpliceSpark");
    kafkaParams.put("group.id", "mynewgroup");
    kafkaParams.put("auto.offset.reset", "earliest");
    kafkaParams.put("enable.auto.commit", false);
    kafkaParams.put("key.deserializer", StringDeserializer.class);
    kafkaParams.put("value.deserializer", StringDeserializer.class);

    List<TopicPartition> topicPartitions= new ArrayList<TopicPartition>();
    for(int i=0; i<5; i++) {
        topicPartitions.add(new TopicPartition("mytopic", i));
    }


    //List of kafka topics to process
    Collection<String> topics = Arrays.asList(this.getTopicList().split(","));


    JavaInputDStream<ConsumerRecord<String, String>> messages = KafkaUtils.createDirectStream(
            jssc,
            LocationStrategies.PreferConsistent(),
            ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)
          );

    //Another version of an attempt
    /*
    JavaInputDStream<ConsumerRecord<String, String>> messages = KafkaUtils.createDirectStream(
        jssc,
        LocationStrategies.PreferConsistent(),
        ConsumerStrategies.<String, String>Assign(topicPartitions, kafkaParams)
      );
     */

    messages.foreachRDD(new PrintRDDDetails());


    // Start running the job to receive and transform the data 
    jssc.start();

    //Allows the current thread to wait for the termination of the context by stop() or by an exception
    jssc.awaitTermination();
}

The call method of the PrintRDDDetails has the following:

    public void call(JavaRDD<ConsumerRecord<String, String>> rdd)
        throws Exception {

    LOG.error("--- New RDD with " + rdd.partitions().size()
            + " partitions and " + rdd.count() + " records");

}

What seems to happen is that it only gets the data from one partition. I've confirmed in kafka that there are 5 partitions. When the call method is executed it prints the correct number of partitions, but only prints the records that are in 1 partition - and further processing that I took out of this simplified code - shows that it is only processing 1 partition.

Answer 1

This seems to be problem with Spark 2.1.0 as it's using v0.10.1 of the kafka-clients library (as per the following pull request):

https://github.com/apache/spark/pull/16278

I worked around this by using a newer version of kafka-clients library:

libraryDependencies ++= Seq(
  "org.apache.spark"  %%  "spark-core"                  % sparkVersion,
  "org.apache.spark"  %%  "spark-streaming"             % sparkVersion,
  "org.apache.spark"  %%  "spark-sql"                   % sparkVersion,
  "org.apache.spark"  %%  "spark-streaming-kinesis-asl" % sparkVersion,
  "org.apache.spark"  %  "spark-streaming-kafka-0-10_2.11"  % sparkVersion,
).map(_.exclude("org.apache.kafka", "kafka-clients"))

libraryDependencies += "org.apache.kafka"  %   "kafka-clients" % "0.10.2.0"