[英]How to send a list of objects using avro & kafka
I'm trying to send a list of objects using Avro & Kafka, my main problem is that my list size may change frequently, so I don't know how to build a dynamic Avro schema since as far as I understand Avro schema meant for well-known structure. 我正在尝试使用Avro和Kafka发送对象列表,我的主要问题是列表大小可能会经常更改,所以据我所知,Avro模式的目的是不知道如何构建动态Avro模式众所周知的结构。
Anybody knows how to do that? 有人知道该怎么做吗?
I think that the easiest way is to make a class with list, for example: 我认为最简单的方法是用list创建一个类,例如:
public class AvroObj {
private List<TestObj> list;
public List<TestObj> getList() {
return list;
}
public void setList(List<TestObj> list) {
this.list = list;
}
}
create a schema: 创建一个模式:
Schema schema = ReflectData.get().getSchema(AvroObj.class);
serialize it to bytes using ReflectDatumWriter (i offer it cause with another datum writers you may catch ClassCastException) 使用ReflectDatumWriter将其序列化为字节(我提供它的原因是您可能会捕获ClassCastException的另一个数据编写器)
try(ByteArrayOutputStream out = new ByteArrayOutputStream()) {
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
DatumWriter<AvroObj> writer = new ReflectDatumWriter<>(schema);
writer.write(avroObj, encoder);
encoder.flush();
byte[] bytes = out.toByteArray();
}
then send bytes with kafka producer. 然后与kafka生产者发送字节。
deserialise bytes that consumer received: 消费者收到的反序列化字节:
DatumReader<AvroObj> reader1 = new ReflectDatumReader<AvroObj>(schema);
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
AvroObj decodedAvroObj = reader1.read(null, decoder);
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.