如何解决Kafka Avro反序列化问题

Question

we have a very strange behavior with an application which used kafka avro consumer and producer.

We have a server where a confluent installation is running. All is configured manually so no confluent cli is used.

On a second server there is an application running which uses this confluent installation to receive requests via Kafka topic. This request is translated into a database query and it sends back a reply topic with specific data.

On a third server there is another application there is the application which sends this requests to the application on server number 2 and receives the reply topics.

So to sum up: Server 1 (Confluent installation) Server 2 (Request Consumer and Reply Producer Application) Server 3 (Request Sender and Reply Consumer Application)

I developed a kafka API for us that all are using the consumer and producers with confluent avro serializer and deserializer. When i'm now producing a request topic from my app on Server 3 my app on Server 2 receives the request, translate it into a database query and sends back a reply topic which is different from the request topic. Now my app on Server 3 should receive the test-reply-topic and deserialize it.

My test-reply-topic contains a "union{null, bytes} file" field. So the field is optional, the default value is null and then schema is registered in the registry. Now if the file value is null it's giving me an error:

Error deserializing key/value for partition test-reply-topic-0 at offset 0. If needed, please seek past the record to continue consumption. Cause: Error deserializing Avro message for id 6 Detailed: Found null, expecting bytes

If it is containing bytes it is working.

The strange thing is, that if I try the same from the local computer it is working with file=null and file=some bytes. So I'm using the same kafka broker, the same API and the same values in the producers and consumers. It is requesting the same schema id from the registry on server and local computer.

Maybe somebody has some hints what the problem could be, I searching since days for this problem and don't find a solution.

Many thanks in advance!

Answer 1

Sorry for my late reply. So it turned out, that the application which had the deserialization problems, used old generated java classes. So it used the old schemas, where there was no union type before.