How to fix the deserialization error when merging 2 kstreams topics using leftJoin?

Question

I am new to Kafka. I am working on a personal project where I want to write to 2 different Avro topics and merge them using leftJoin. Once I merge them, I want to produce the same messages to a KSQL DB as well. (I haven't implemented that part yet).

I am using Kafka Template to produce to the 2 Avro topics and convert them into kstreams to merge them. I am also using KafkaListener to print any messages in them and that work is working. Here's where I am having issues at: 2 of them actually. In either cases, it doesn't produce any messages in the merged topic.

1. If I removed the consumed.with() from the kstream, then it throws a default key Serde error.

2. But if I keep it, then it throws a deserialization error.

I have even provided the default serialization and deserialization in both my application.properties and in the streamConfig inside main() but it's still not working.

Can somebody please help me with how to merge the 2 Avro topics? Is it error occurring because I am using the Avro schema? Should I use JSON instead? I wanna use a schema because my value part of the message will have multiple values in it.

For eg: {Key: Value} = {company: {inventory_id, company, color, inventory}} = {Toyota: {0, RAV4, 50,000}}

Here's a link to all the file: application.properties , DefaultKeySerdeError.txt , DeserializationError.txt , FilterStreams.java , Inventory.avsc , Pricing.avsc , and MergedAvro.avsc . Let me know if yall want me to put them below. Thank you very much for your help in advance!

https://gist.github.com/Arjun13/b76f53c9c2b4e88225ef71a18eb08e2f

Answer 1

Looking at the DeserializationError.txt file, it looks like the problem is you haven't provided the credentials for schema registry. Even though you have provided them in the application.properties file, they're not getting into the serdes configuration , so if you add the basic.auth.user.info configs to the serdeConfig map you should be all set.