Kafka Avro 控制台消费者/生产者从文件到文件

Question

Is it possible to save Avro encoded messages to a file using Avro Console Consumer and reading from this file to write to another topic using Avro Console Producer given I have a schema string available to provide to the console consumer and producer? Is this a use cases that are supported out of box or do I need to write a shell script for it?

Answer 1

The Avro console producer only accepts JSON encoded strings, not Avro files. However, given an Avro file, you can dump its records as JSON and AVSC schema, then it's possible to parse into the console producer (however, it doesn't work in all cases, in my experience).

You can use Kafka Connect S3 or HDFS sinks (or Apache Nifi) to consume Schema Registry encoded data and write into local Avro files.

You can MinIO or Hadoop Ozone to emulate a local S3 endpoint, or you can use the file:// URI prefix with the HDFS connector to write to local disk

There's no HDFS source Kafka connector that I know of, but Confluent does provide an S3 source connector. Nifi can read and write both file locations, so maybe start with it

Flink or Spark might also work, but have less straightforward Avro serializer settings to make it back into Schema Registry format