![](/img/trans.png)
[英]Pulsar function fails to deserialize message because of wrong schema type (JSON instead of AVRO)
[英]Avro fails to deserialize message with updated schema
我有一個已更新為包含新字段的架構。 我正在使用avro反射和匯合模式注冊表來反序列化/序列化數據,如下所示:
連載:
Schema schema = REFLECT_DATA.getSchema(value.getClass());
try {
int registeredSchemaId = this.schemaRegistry.register(subject, schema);
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(0);
out.write(ByteBuffer.allocate(4).putInt(registeredSchemaId).array());
DatumWriter<Object> dw = new ReflectDatumWriter<>(schema);
Encoder encoder = ENCODER_FACTORY.directBinaryEncoder(out, null);
dw.write(value, encoder);
encoder.flush();
return out.toByteArray();
} catch (RuntimeException | IOException e) {
throw new SerializationException("Error serializing Avro message", e);
} catch (RestClientException e) {
throw new SerializationException("Error registering Avro schema: " + schema, e);
}
反序列化:
if (readerSchema == null) {
readerSchema = new Schema.Parser().parse(schemaString);
}
int schemaId = -1;
try {
ByteBuffer buffer = ByteBuffer.wrap(payload);
if (buffer.get() != MAGIC_BYTE) {
throw new SerializationException("Unknown magic byte!");
}
schemaId = buffer.getInt();
Schema writerSchema = schemaRegistry.getById(schemaId);
int start = buffer.position() + buffer.arrayOffset();
int length = buffer.limit() - 1 - idSize;
DatumReader<Object> reader = new ReflectDatumReader<>(writerSchema, readerSchema);
BinaryDecoder decoder = decoderFactory.binaryDecoder(buffer.array(), start, length, null);
return reader.read(null, decoder); //line 83
} catch (IOException e) {
throw new SerializationException("Error deserializing Avro message for id " + schemaId, e);
} catch (RestClientException e) {
throw new SerializationException("Error retrieving Avro schema for id " + schemaId, e);
}
模式由scala案例類定義,舊模式如下所示:
case class Data(oldField: String) {
def this("")
}
並且它已經像這樣更新:
case class Data(oldField: String, @AvroDefault("") newField: String) {
def this("", "")
}
但是,反序列化有時會拋出具有以下堆棧的AvroTypeException:
Caused by: org.apache.avro.AvroTypeException: Found com.company.project.DataHolder$.Data, expecting com.company.project.DataHolder$.Data
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
at io.fama.pubsub.KafkaAvroReflectDeserializer.deserialize(KafkaAvroReflectDeserializer.java:83)
我認為這是由序列化舊消息的困難引起的(但我不完全確定 - 我無法推斷它可能是什么)。 有沒有其他人遇到過這個錯誤或有沒有人有任何想法來解決它?
如果您正在使用org.apache.avro.reflect屬性,那么我認為您不能使用Scala案例類 - Scala案例類參數是不可變的,我相信屬性映射器需要有一個公共類空構造函數和java可見字段,甚至可能是@BeanProperty來生成java setter / getters。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.