Spark Streaming Unit Test in Java

Question

I am trying write Unit test for my Spark streaming Job. My spark streaming job consumes messages from MQ and push it in to the kafka topic.

My approach is

• Send a Test Msg to MQ
• Start a Streaming job in a separate thread. (Streaming job will push the messages to kafka topic " topic1 ")
• kafka consumer to keep polling the topic1 .
• Once the message recieved, stop the thread and break from the loop.

Below is my code and its not working. Spark streaming job starting fine, but once the streaming job started, my while loop stopped looping. Not sure about the reason since I am new to the Concurrency topic

public class StreamingJobTest {

private static KafkaConsumer<String, String> consumer;

@BeforeClass
public static void setUpClass()  {

    Properties properties = new Properties();

    properties.put("bootstrap.servers", "localhost:9090");
    properties.put("subscribe", "topic1");
    properties.put("startingOffsets", "earliest");
    properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

    consumer = new KafkaConsumer<String, String>(properties);



}


@Test
public void create_test() {
    String[] arguments = new String[]{};
    ConsumerRecords<String, String> records;

    Thread thread = new Thread(() -> StreamingJob.main(arguments));
    thread.start();

     //send a message to MQ.

    MqSender mqSender = new MqSender();
    mqSender.mqPushMsg("TestMsg");

    //keep polling the kafka topic.

    while(true){
        System.out.println("Polling...");
        records = consumer.poll(100);

        if(!records.isEmpty()){

            thread.interrupt();
            break;
        }

    assertNotNull(records);

    }


}

}

Why my loop stopped working after the streaming job starts? As per my understanding streaming will be running in separate thread right?

Answer 1

I figured out myself. I need to subscribe the topic in a separate line. I added it to my properties. And also groupid is mandatory in kafka and I missed it. Its works fine for me now. below is the code to subscribe the topic.

consumer.subscribe(Arrays.asList("topic1"));