How to move past undeserializable messages in Spring Kafka 1.3 with Avro

Question

I can't upgrade to Spring 5 so I'm stuck with spring-kafka 1.3 and its limited error handling. So I have no access to ConsumerAwareErrorHandler or SeekToCurrent error handlers from spring-kafka 2.

I'm using a @KafkaListener -annotated method to listen to a topic, I've configured the io.confluent.kafka.serializers.KafkaAvroDeserializer as my value deserializer.

The problem is if I end up with a message in my topic that is not in Avro format the KafkaMessageListenerContainer poll loop gets stuck. The deserializer throws an exception on the message and the poll loop never seeks past it so the next time through the loop it tries deserializing the same message and keeps looping, dumping the same error thousands of time per second into my log.

There doesn't seem to be a way to get a NeverRetryPolicy or anything in edge-wise, but I can factory.getContainerProperties().setErrorHandler() . Unfortunately I'm not sure what I can do from there.

Is there something I can Autowire into my error handler that I can use to seek forward 1 offset on error? Not sure what that is, the docs don't talk much about what you can actually DO with the ErrorHandler and most examples I can find are for spring-kafka 2.X. Like, it won't deserialize, there's NOTHING I can do with the message, it will NEVER work, I want to avoid retrying it ever again and it seems most Stackoverflow questions are about doing the opposite.

I've also seen some folks just wrap the Deserializer from Avro with their own class that eats the exception and returns null. Is that a better plan?

Answer 1

The problem is the deserialization fails well before Spring gets the data - the problem is in Kafka itself.

In 2.2, we added the ErrorHandlingDeserializer2 which wraps the real deserializer and sends a signal to the listener container so the error can be sent to the error handler.

In older versions, you would need to write your own deserializer wrapper - however, there is no code in the container to deal with the situation so your catch block would need to return a real Object that is a signal to your listener that the deserialization failed.

Let's say your listener receives Invoice . Your catch block could create a subclass, say BadInvoice which you can then detect in your listener and discard it.

I've also seen some folks just wrap the Deserializer from Avro with their own class that eats the exception and returns null. Is that a better plan?

You can return null if you never get real null records, but you would have to add @Payload(required = false) to the method parameter.