Validate Kafka messages using Schema Registry API

Question

I'm implementing a process to produce kafka messages and every message should have the schema validated by Schema Registry. For development, I'm running kafka and Schema Registry using docker and my schemas are being registed by schema registry ui.

It looks like that my schemas are not being validated or I am missing some configuration. My producer class has the following code:

package br.com.xx.realtime_transformation.producers;

import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;

import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.avro.generic.GenericRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.UUID;

public class KafkaEventProducer {

    private final String bootstrapServers;
    private final String schemaRegistryUrl;
    private final KafkaProducer<String, GenericRecord> kafkaProducer;

    public KafkaEventProducer(String bootstrapServers, String schemaRegistryUrl) {
        this.bootstrapServers = bootstrapServers;
        this.schemaRegistryUrl = schemaRegistryUrl;
        this.kafkaProducer = getProducer();
    }

    private KafkaProducer<String, GenericRecord> getProducer() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.bootstrapServers);
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.RETRIES_CONFIG, 0);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        props.put(AbstractKafkaAvroSerDeConfig.AUTO_REGISTER_SCHEMAS, false);
        props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, this.schemaRegistryUrl);

        return new KafkaProducer<>(props);
    }

    public void send(String topic, GenericRecord event) {
        try {
            String key = UUID.randomUUID().toString();
            final ProducerRecord producerRecord = new ProducerRecord<>(topic, key, event);

            this.kafkaProducer.send(producerRecord);
        } catch (final SerializationException e) {
            e.printStackTrace();
        }

    }

}

Most of times I get an error like "Schema not found" and when this exception isn't thrown, my message is not validated, it just send the message to another topic.

Is there missing any kind of configuration?

Answer 1

Kafka Producer works with schema Registry and without schema Registry.

Without Schema Registry - Following is the example of Kafka Producer Without schema Registry

https://www.devglan.com/apache-kafka/apache-kafka-java-example

With Schema Registry

The schema.registry.url parameter simply points to where we store the schemas. We need to provide the schema definition as shown below. You are not providing a schema definition but using the schema registry so you are getting the issue.

{
  "namespace": "com.example",
  "type": "record",
  "name": "Employee",
  "doc" : "Represents an Employee at a company",
  "fields": [
      {"name": "firstName", "type": "string", "doc": "The persons given name"},
      {"name": "lastName", "type": "string"},
      {"name": "age",  "type": "int", "default": -1},
      {"name": "emails", "default":[], "type":{"type": "array", "items": "string"}},
      {"name": "phoneNumber",  "type": "string"}
   ]
} 

The workflow of Schema Registry is shown below.

You can find a detailed explanation at the following URL. https://mapr.com/docs/61/Kafka/KafkaSchemaRegistry/KafkaSchemaRegistryDemo.html https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html https://www.confluent.jp/blog/kafka-connect-tutorial-transfer-avro-schemas-across-schema-registry-clusters/

Answer 2

I'm not sure I understand the question, but you seem to be asking about Schema Validation.

This is being handled in the 5.4 release