FlinkKafkaConsumer / KafkaSource with AWS Glue Schema Registry or Confluent Schema Registry

Question

I'm trying to write an Flink streaming application that has a KafkaSource to read from a topic which has an AVRO schema defined for its data.

I would like to know how the automatic caching of schemas locally works in this case similar to Confluent's documentation here .

Basically, the use-case is that a consumer should not know the schema beforehand. Once the consumer is instantiated, the schema registry URL should be taken as a parameter and the consumer should read the schema for that particular topic.

Is this possible? Any pointers are appreciated!

Answer 1

Once the consumer is instantiated, the schema registry URL should be taken as a parameter and the consumer should read the schema for that particular topic.

It will, and that will be cached. This is known as the "writer schema".

consumer should not know the schema beforehand

It needs to since Avro requires a "reader schema" to deserialize data defined by the "writer schema".

Without a reader schema, you are left with handling Avro GenericRecord types

Answer 2

The AWS SerDe libraries for Glue use a wire format that containes the uuid of the schema (version) the message is serialized with. The consuming application reads the schema id from the message, and loads it from the Glue schema registry, if it's not in the local cache already. You can find a description of the wire format at the bottom of the readme for this javascript serde library: https://github.com/meinestadt/glue-schema-registry .

Answer 3

This should be possible.

You can test with a Kafka CLI tool like kcat like this:

kcat -b mybroker -t ledger -s avro -r http://schema-registry-url:8080

If you are using kafka-avro-console-consumer :

kafka-avro-console-consumer --topic topicX --bootstrap-server kafka:9092 \ --property schema.registry.url="http://schema-registry:8081"