kafka java consumer not reading data

Question

I am trying to write a simple java kafka consumer to read data using similar code as in https://github.com/bkimminich/apache-kafka-book-examples/blob/master/src/test/kafka/consumer/SimpleHLConsumer.java .

Looks like my app is able to connect, but its not fetching any data. Please suggest.

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
//import scala.util.parsing.json.JSONObject
import scala.util.parsing.json.JSONObject;

public class SimpleHLConsumer {

    private final ConsumerConnector consumer;
    private final String topic;

    public SimpleHLConsumer(String zookeeper, String groupId, String topic) {
        Properties props = new Properties();
        props.put("zookeeper.connect", zookeeper);
        props.put("group.id", groupId);
      //  props.put("zookeeper.session.timeout.ms", "5000");
      //  props.put("zookeeper.sync.time.ms", "250");
      //  props.put("auto.commit.interval.ms", "1000");

        consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
        this.topic = topic;
    }

    public void testConsumer() {
        Map<String, Integer> topicCount = new HashMap<>();
        topicCount.put(topic, 1);

        Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
        System.out.println(consumerStreams);
        List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
        System.out.println(streams);
        System.out.println(consumer);
        for (final KafkaStream stream : streams) {
            ConsumerIterator<byte[], byte[]> it = stream.iterator();
            System.out.println("for loop");
            System.out.println(it);
            System.out.println("Message from Single Topic: " + new String(it.next().message()));
            //System.out.println("Message from Single Topic: " + new String(it.message()));
            while (it.hasNext()) {
                System.out.println("in While");
                System.out.println("Message from Single Topic: " + new String(it.next().message()));
            }
        }
       // if (consumer != null) {
       //     consumer.shutdown();
       // }
    }

    public static void main(String[] args) {
        String topic = "test";
        SimpleHLConsumer simpleHLConsumer = new SimpleHLConsumer("localhost:2181", "testgroup", topic);
        simpleHLConsumer.testConsumer();
    }

}

Here is the output i see in eclipse. It does seem to connect to my zookeeper , but it just hangs there, it does not display any message at all.

log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
SLF4J: The requested version 1.6 by your slf4j binding is not compatible with [1.5.5, 1.5.6]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.
{test=[testgroup kafka stream]}
[testgroup kafka stream]
kafka.javaapi.consumer.ZookeeperConsumerConnector@6200f9cb
for loop

Answer 1

Consumer iterator hasNext is blocking call. It will block indefinitely if no new message is available for consumption.

To verify this, change your code to

// Comment 2 lines below
// System.out.println(it);
// System.out.println("Message from Single Topic: " + new String(it.next().message()));
// Line below is blocking. Your code will hang till next message in topic. 
// Add new message in topic using producer, message will appear in console
 while (it.hasNext()) {

Better way is to execute code in separate thread. Use consumer.timeout.ms to specify time in ms, after which consumer will throw timeout exception

// keepRunningThread is flag to control when to exit consumer loop
while(keepRunningThread) 
{
  try  
  {
    if(it.hasNext())
    {
        System.out.println(new String(it.next().message()));
    }
  } 
  catch(ConsumerTimeoutException ex) 
  {
    // Timeout exception waiting for kafka message
    // Wait for 5 (or t) seconds before checking for message again
     Thread.sleep(5000);
  }
}‍‍‍‍‍‍‍‍‍‍