Flink reads avro topic with AvroDeserializationSchema

Question

I am trying to read avro kafka topic in Scala with case class:

val senv = StreamExecutionEnvironment.getExecutionEnvironment

val kafkaSourceTopic = "device_logs"
val kafkaBrokers = "localhost:9092"

val properties = new Properties()
properties.setProperty("bootstrap.servers", kafkaBrokers)
properties.setProperty("group.id", "flink_app_avro")

case class DeviceData(
  custom_fields: String,
  upload_time: Long,
  build: String,
  device_id: String,
  log_version: Int
)

val consumer = new FlinkKafkaConsumer[DeviceData](kafkaSourceTopic, new AvroDeserializationSchema[DeviceData](classOf[DeviceData]), properties)
val stream = senv.addSource(consumer)

Got this error when running the code in flink-scala-shell:

scala> val consumer = new FlinkKafkaConsumer[DeviceData](kafkaSourceTopic, new AvroDeserializationSchema[DeviceData](classOf[DeviceData], null), properties)
<console>:92: error: constructor AvroDeserializationSchema in class AvroDeserializationSchema cannot be accessed in object $iw
       val consumer = new FlinkKafkaConsumer[DeviceData](kafkaSourceTopic, new AvroDeserializationSchema[DeviceData](classOf[DeviceData], null), properties)

How to read avro topic with AvroDeserializationSchema in Scala? Thanks.

Answer 1

You can't simply create a case class and tell Flink to parse avro to that automatically. Avro is a binary format and You need to know schema which You will use to parse the records. Due to that, You can either:

1. Get the schema of records from Kafka, and use that to create AvroDeserializationSchema for GenericRecord using AvroDeserializationSchema.forGeneric .
2. You can take a look at Avro plugins for the build tool You are using and then create a file with Avro schema of records from Kafka and use Avro plugin to generate the classes for them. Then You can simply use `AvroDeserializationSchema.forSpecific([nameOfTheGeneratedClass])