Scala And Avro: Convert the case class to avro record

Question

I am using Scala 2.12 and Avro (org.apache.avro) 1.8.

I have the following schema:

Schema: {"name": "person","type": "record","fields": [{"name": "address","type": {"type" : "record","name" : "AddressUSRecord","fields" : [{"name": "streetaddress", "type": "string"},{"name": "city", "type":"string"}]}}]}

Corresponding Scala case classes are:

case class AddressUSRecord (streetaddress: String, name: String}

case class Address (addressUSRecord: List[AddressUSRecord])

case class Person (person: Address)

I am using GenericRecord to convert my object of case class PnlRecord into Avro.

val schema = new Schema.Parser().parse(new File(schemaFileName))
val avroRecord = new GenericData.Record(schema)
val writer = new GenericDatumWriter[GenericRecord](schema)
val out = new ByteArrayOutputStream()
val encoder = EncoderFactory.get().binaryEncoder(out, null)
val producer = new KafkaProducer[String, Array[Byte]](properties)
avroRecord.put("header", record.header)
//Please note that this pnlData (see above case class) is complex and created accordingly.
avroRecord.put("pnlData", record.pnlData)
writer.write(avroRecord, encoder)
val bytes = out.toByteArray
encoder.flush()
out.close()

I am getting the following error.

2019-03-13 21:57:29.832 [application-akka.actor.default-dispatcher-4] ERROR controllers.SAController.$anonfun$publishToSA$2(34) - ca.company.project.sa.model.MessageHeader cannot be cast to org.apache.avro.generic.IndexedRecord
java.lang.ClassCastException: ca.company.project.sa.model.MessageHeader cannot be cast to org.apache.avro.generic.IndexedRecord
        at org.apache.avro.generic.GenericData.getField(GenericData.java:697)
        at org.apache.avro.generic.GenericData.getField(GenericData.java:712)
        at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:164)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
        at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
        at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
        at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)

Why my MessageHeader case class cannot be cast to IndexedRecord? What am I missing here?

How do we convert such complex case class to avro object? Can someone help with such nested case class example to convert to avro record?

Thanks in advance.

Answer 1

Confluent Kafka Avro serializer is Java-based, and, as such, is most likely not designed to work with Scala objects. I see that your pnlBreakdown is declared as List[PnlBreakdown] - if this is a Scala list, the serializer won't even recognize it as a collection. Same goes for case classes - these won't be recognized as Java Beans without the @BeanProperty annotations