Spring-Cloud-Stream Kafka Azure-提取数据时出现意外错误代码13

Question

I'm developing a SpringBoot app dockerized. The docker images are microservices and one of these communicate with Azure Event Hub.

Some of my properties:

spring-boot -> 2.0.7.RELEASE

spring-cloud.version -> Finchley.SR2

I've created a topic in Azure(with Kafka enabled ).

I've follow some simple guide to set up my microservice and everything works fine.

@EnableBinding({Processor.class})
public class EventService {
    ...
    @Autowired private Processor ehProcessor;
    ...
    public void send(String event) {

        Message<String> message = MessageBuilder
                .withPayload(event)
                .setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
                .build();

        boolean send = ehProcessor.output().send(message, 5000L);

        if (!send) {

            log.error("Event NOT sent", event);
        }
    }

    ...

    @StreamListener(target = Processor.INPUT)
    public void receive(String event) {

        handle(event);
    }
}

For an entire month everything goes fine but in the last two days the microservice stucks because a continuous stacktrace is filling all my disc (a solution was to set up a docker log rotation).

java.lang.IllegalStateException: Unexpected error code 13 while fetching data
        at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:891) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:528) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1154) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1111) ~[kafka-clients-1.0.1.jar!/:na]
        at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:699) ~[spring-kafka-2.1.7.RELEASE.jar!/:2.1.7.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

I'm talking about 8/9 log messsage in 1 ms.

Looking inside the org.apache.kafka.common.protocol.Errors class the error is related to:

NETWORK_EXCEPTION(13, "The server disconnected before a response was received."

I'm not able to reproduce programmatically this error. I don't understand why when the first error is arised the log will start and no stop in infinite loop. I need to stop the docker container and sometimes the container will not stop. The only solution is to remove the container and recreate again.

UPDATE

I've open an issue on github here . I've already received a response and they are starting to investigate on it.

UPDATE

The problem has been fixed.

When they changed an UnknownServerException to a NetworkException, Spring Boot started getting stuck in the retry loop.

Answer 1

It has confirm in this link that,

There was a recent change where an instance of UnknownServerException was changed to a NetworkException.

Issue details are here - https://github.com/Azure/azure-event-hubs-for-kafka/issues - with your namespace info. Thanks!