[英]Validate Kafka messages using Schema Registry API
I'm implementing a process to produce kafka messages and every message should have the schema validated by Schema Registry.我正在实现一个生成 kafka 消息的过程,并且每条消息都应该具有由 Schema Registry 验证的模式。 For development, I'm running kafka and Schema Registry using docker and my schemas are being registed by schema registry ui.对于开发,我正在使用 docker 运行 kafka 和模式注册表,并且我的模式正在由模式注册表 ui 注册。
It looks like that my schemas are not being validated or I am missing some configuration.看起来我的模式没有得到验证,或者我缺少一些配置。 My producer class has the following code:我的制片人 class 有以下代码:
package br.com.xx.realtime_transformation.producers;
import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.avro.generic.GenericRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.UUID;
public class KafkaEventProducer {
private final String bootstrapServers;
private final String schemaRegistryUrl;
private final KafkaProducer<String, GenericRecord> kafkaProducer;
public KafkaEventProducer(String bootstrapServers, String schemaRegistryUrl) {
this.bootstrapServers = bootstrapServers;
this.schemaRegistryUrl = schemaRegistryUrl;
this.kafkaProducer = getProducer();
}
private KafkaProducer<String, GenericRecord> getProducer() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.bootstrapServers);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, 0);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(AbstractKafkaAvroSerDeConfig.AUTO_REGISTER_SCHEMAS, false);
props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, this.schemaRegistryUrl);
return new KafkaProducer<>(props);
}
public void send(String topic, GenericRecord event) {
try {
String key = UUID.randomUUID().toString();
final ProducerRecord producerRecord = new ProducerRecord<>(topic, key, event);
this.kafkaProducer.send(producerRecord);
} catch (final SerializationException e) {
e.printStackTrace();
}
}
}
Most of times I get an error like "Schema not found" and when this exception isn't thrown, my message is not validated, it just send the message to another topic.大多数情况下,我会收到“找不到架构”之类的错误,并且当未引发此异常时,我的消息未经过验证,它只是将消息发送到另一个主题。
Is there missing any kind of configuration?是否缺少任何类型的配置?
Kafka Producer works with schema Registry and without schema Registry. Kafka Producer 使用模式注册表和没有模式注册表。
Without Schema Registry - Following is the example of Kafka Producer Without schema Registry没有模式注册表- 以下是没有模式注册表的 Kafka Producer 的示例
https://www.devglan.com/apache-kafka/apache-kafka-java-example https://www.devglan.com/apache-kafka/apache-kafka-java-example
With Schema Registry使用模式注册表
The schema.registry.url
parameter simply points to where we store the schemas. schema.registry.url
参数只是指向我们存储模式的位置。 We need to provide the schema definition as shown below.我们需要提供如下所示的模式定义。 You are not providing a schema definition but using the schema registry so you are getting the issue.您没有提供架构定义,而是使用架构注册表,因此您遇到了问题。
{
"namespace": "com.example",
"type": "record",
"name": "Employee",
"doc" : "Represents an Employee at a company",
"fields": [
{"name": "firstName", "type": "string", "doc": "The persons given name"},
{"name": "lastName", "type": "string"},
{"name": "age", "type": "int", "default": -1},
{"name": "emails", "default":[], "type":{"type": "array", "items": "string"}},
{"name": "phoneNumber", "type": "string"}
]
}
The workflow of Schema Registry is shown below. Schema Registry 的工作流程如下所示。
You can find a detailed explanation at the following URL.你可以在下面的URL找到详细的解释。 https://mapr.com/docs/61/Kafka/KafkaSchemaRegistry/KafkaSchemaRegistryDemo.html https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html https://www.confluent.jp/blog/kafka-connect-tutorial-transfer-avro-schemas-across-schema-registry-clusters/ https://mapr.com/docs/61/Kafka/KafkaSchemaRegistry/KafkaSchemaRegistryDemo.html https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html https://www.confluent.jp/blog/kafka-connect-tutorial-transfer-avro-schemas-across-schema-registry-clusters/
I'm not sure I understand the question, but you seem to be asking about Schema Validation.我不确定我是否理解这个问题,但您似乎在询问模式验证。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.