Spring Boot Kafka应用程序未收到消息

Question

We're using Spring boot 1.5.4.RELEASE and i can see the following jars in our gradle project: spring-kafka-1.1.6, kafka-clients-0.10.1.1

We have a simple receiver class like below:

`

   @KafkaListener(topics = "${mytopic.consume}")
    public void receive(byte[] event) {
        <our code here>...
    }

//$mytopic.consume is being picked up from properties file

`

The code works fine on most days and i am able to receive events correctly and process them, then suddenly it stops to receive messages. No error, no warnings. I can see the messages are there on the topic but our listener just doesn't seem to consume them. I tried restarting application, restarting eclipse, change between run and debug mode, change the name of group in kafka properties - but nothing works. We look at the application after few hours, and there you go - it starts to work again and we receive all the messages which were stuck. Sometimes the problem can last more than a day also. My suspicion is its NOT an eclipse issue, but probably the way enterprise Kafka is setup or some bug in Kafka(lesser possibility since i couldn't find such an issue on google). I did notice this issue even when microservice was deployed on IBM bluemix.

Any pointers will be greatly appreciated.

Here are some of our properties(which might be of interest) that we have configured in our application: `

acks=all
     retries=0
     batch.size=<a number greater than 10k>
     linger.ms=1
     buffer.memory=<a number in 10s of MBs - our incoming message is just few KBs>
     ENABLE_AUTO_COMMIT_CONFIG=true
     AUTO_COMMIT_INTERVAL_MS_CONFIG=1ms
     SESSION_TIMEOUT_MS_CONFIG=30ms
     zookeeper.sync.time=200ms
     zookeeper.session.timeout=400ms

`

Answer 1

There is nothing blatantly wrong with the configurations. I would advise you to step back and start over again. Create a new Consumer group. Potentially, also write the code from scratch.

Do tests having different number of consumers in a consumer group, note consumers should be equal to or less than number of partitions. Do tests, with different amount of load of messages and see how it works out. In the mean while, keep an eye on the Kafka Cluster also, do any brokers leave the cluster, which would change the leaders for the partitions. Eventually, you can stop some consumers, and see how rebalance affects your consumers .

Hope this helps.