Sending Avro messages to Kafka

Question

I have an app that produces an array of messages in raw JSON periodically. I was able to convert that to Avro using the avro-tools. I did that because I needed the messages to include schema due to the limitations of Kafka-Connect JDBC sink. I can open this file on notepad++ and see that it includes the schema and a few lines of data.

Now I would like to send this to my central Kafka Broker and then use Kafka Connect JDBC sink to put the data in a database. I am having a hard time understanding how I should be sending these Avro files I have to my Kafka Broker. Do I need a schema registry for my purposes? I believe Kafkacat does not support Avro so I suppose I will have to stick with the kafka-producer.sh that comes with the Kafka installation (please correct me if I am wrong).

Question is: Can someone please share the steps to produce my Avro file to a Kafka broker without getting Confluent getting involved.

Thanks,

Answer 1

To use the Kafka Connect JDBC Sink, your data needs an explicit schema. The converter that you specify in your connector configuration determines where the schema is held. This can either be embedded within the JSON message ( org.apache.kafka.connect.json.JsonConverter with schemas.enabled=true ) or held in the Schema Registry (one of io.confluent.connect.avro.AvroConverter , io.confluent.connect.protobuf.ProtobufConverter , or io.confluent.connect.json.JsonSchemaConverter ).

To learn more about this see https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained

To write an Avro message to Kafka you should serialise it as Avro and store the schema in the Schema Registry. There is a Go client library to use with examples

without getting Confluent getting involved.

It's not entirely clear what you mean by this. The Kafka Connect JDBC Sink is written by Confluent. The best way to manage schemas is with the Schema Registry. If you don't want to use the Schema Registry then you can embed the schema in your JSON message but it's a suboptimal way of doing things.