Issue with Spring Cloud Stream using the Default Schema Registry client instead of the Avro Schema Registry client

Question

We are using Kafka with Spring Cloud Stream and we need to connect to a Confluent Schema Registry in our Spring Boot component see https://github.com/donalthurley/KafkaConsumeScsAndConfluent .

We have added the following configuration to create the required ConfluentSchemaRegistryClient bean see https://github.com/donalthurley/KafkaConsumeScsAndConfluent/blob/master/src/main/java/com/example/kafka/KafkaConfig.java which should override the default schema registry from Spring Cloud Stream.

However we have been seeing the following failure intermittently after some deployments.

org.springframework.messaging.MessageDeliveryException: failed to send Message to channel

The underlying cause shows this stack trace

Caused by: java.lang.NullPointerException
    at     org.springframework.cloud.stream.schema.client.DefaultSchemaRegistryClient.register(DefaultSchemaRegistryClient.java:71)
    at org.springframework.cloud.stream.schema.avro.AvroSchemaRegistryClientMessageConverter.resolveSchemaForWriting(AvroSchemaRegistryClientMessageConverter.java:238)
    at org.springframework.cloud.stream.schema.avro.AbstractAvroMessageConverter.convertToInternal(AbstractAvroMessageConverter.java:179)
    at org.springframework.messaging.converter.AbstractMessageConverter.toMessage(AbstractMessageConverter.java:201)
    at org.springframework.messaging.converter.AbstractMessageConverter.toMessage(AbstractMessageConverter.java:191)
    at org.springframework.messaging.converter.CompositeMessageConverter.toMessage(CompositeMessageConverter.java:83)
    at org.springframework.cloud.stream.binding.MessageConverterConfigurer$OutboundContentTypeConvertingInterceptor.doPreSend(MessageConverterConfigurer.java:322)
    at org.springframework.cloud.stream.binding.MessageConverterConfigurer$AbstractContentTypeInterceptor.preSend(MessageConverterConfigurer.java:351)
    at org.springframework.integration.channel.AbstractMessageChannel$ChannelInterceptorList.preSend(AbstractMessageChannel.java:611)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453)

The fact that the DefaultSchemaRegistryClient is being invoked by the AvroSchemaRegistryClientMessageConverter would indicate to us that there is a problem with the wiring of our ConfluentSchemaRegistryClient bean.

Is there something else required in our configuration to ensure the ConfluentSchemaRegistryClient bean is wired correctly?

Answer 1

It worked for me. This is what I did :

• used the configuration class exactly like you did
• used the @EnableSchemaRegistryClient annotation in the project
• added the avro serializer to the classpath : io.confluent:kafka-avro-serializer

• set the properties as follows:

  spring.cloud.stream.kafka.bindings.channel.consumer.configuration.schema.registry.url=address of your registry spring.cloud.stream.kafka.bindings.channel.consumer.configuration.specific.avro.reader=true

where channel corresponds to the channel name in your app.

I think the last property is very important to tell Spring to use Avro serializer instead of the default serializer.

I am using Spring Cloud Stream Elmhurst.RELEASE, so properties' names might slightly differ if you use another version.

Answer 2

I have now moved the @EnableSchemaRegistryClient annotation to the project application class see https://github.com/donalthurley/KafkaConsumeScsAndConfluent/commit/b4cf5427d7ab0a4fed619fe54b042890f5ccb594 and redeployed and this has fixed the issue I had when it was deployed on our environment.

I had been annotating my producer and consumer classes with the @EnableSchemaRegistryClient annotation.

In all my local testing this had been working against my local docker confluent schema registry. However on deploying to our environments it was working most of the time but failed occasionally after some deploys.

I hadn't succeeded in replicating this locally.

I noticed also in testing locally that I get the same null pointer exception stack trace if I remove the configuration for the Confluent Schema Registry.

So the problem I think I was seeing is that the AvroSchemaRegistryClientMessageConverter bean was not wired with the confluent schema registry bean when the @EnableSchemaRegistryClient annotation was not present in the project application class.

I don't understand why exactly that would be necessary but I think it may have solved the issue.