[英]Kafka Protobuf: C++ serialization to java
I've developed a couple of C++ apps that produce and consume Kafka messages (using cppkafka) embedding Protobuf3 messages. 我开发了几个C ++应用程序来生成和使用Kafka消息(使用cppkafka)嵌入Protobuf3消息。 Both work fine.
两者都很好。 The producer's relevant code is:
制片人的相关代码是:
std::string kafkaString;
cppkafka::MessageBuilder *builder;
...
solidList->SerializeToString(&kafkaString);
builder->payload(kafkaString);
Protobuf objects are serialized to string and inserted as Kafka payload. Protobuf对象被序列化为字符串并作为Kafka有效负载插入。 Everything works fine up to this point.
到目前为止,一切正常。 Now, I'm trying to develop a consumer for that in Java.
现在,我正在尝试用Java开发一个消费者。 The relevant code should be:
相关代码应为:
KafkaConsumer<Long, String> consumer=new KafkaConsumer<Long, String>(properties);
....
ConsumerRecords<Long, String> records = consumer.poll(100);
for (ConsumerRecord<Long, String> record : records) {
SolidList solidList = SolidList.parseFrom(record.value());
...
but that fails at compile time: parseFrom complains: The method parseFrom(ByteBuffer) in the type Solidlist.SolidList is not applicable for the arguments (String) . 但是在编译时失败了:parseFrom抱怨: Solidlist.SolidList类型中的方法parseFrom(ByteBuffer)不适用于参数(String) 。 So, I try using a ByteBuffer:
所以,我尝试使用ByteBuffer:
KafkaConsumer<Long, ByteBuffer> consumer=new KafkaConsumer<Long, ByteBuffer>(properties);
....
ConsumerRecords<Long, ByteBuffer> records = consumer.poll(100);
for (ConsumerRecord<Long, ByteBuffer> record : records) {
SolidList solidList = SolidList.parseFrom(record.value());
...
Now, the error is on execution time, still on parseFrom(): Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.nio.ByteBuffer . 现在,错误是执行时间,仍然在parseFrom()上: 线程“main”中的异常java.lang.ClassCastException:java.lang.String无法强制转换为java.nio.ByteBuffer 。 I know it is a java.lang.String!!!
我知道这是一个java.lang.String !!! So, I get back to the original, and try using it as a byte array:
所以,我回到原来的,并尝试将其用作字节数组:
SolidList solidList = SolidList.parseFrom(record.value().getBytes());
Now, the error is on execution time: Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type. 现在,错误在于执行时间: 线程“main”中的异常com.google.protobuf.InvalidProtocolBufferException $ InvalidWireTypeException:协议消息标记具有无效的线路类型。 .
。
The protobuf documentation states for the C++ serialization: bool SerializeToString(string output) const;: serializes the message and stores the bytes in the given string. protobuf文档说明了C ++序列化: bool SerializeToString(string output)const;:序列化消息并将字节存储在给定的字符串中。 Note that the bytes are binary, not text ;
请注意,字节是二进制的,而不是文本 ; we only use the string class as a convenient container.*
我们只使用字符串类作为方便的容器。*
TL;DR: In consequence, how should I interpret the protobuf C++ "binary bytes" in Java? TL; DR:因此,我应该如何解释Java中的protobuf C ++“二进制字节”?
This seems related (it is the opposite) but doesn't help: Protobuf Java To C++ Serialization [Binary] 这似乎是相关的(它是相反的)但没有帮助: Protobuf Java到C ++序列化[二进制]
Thanks in advance. 提前致谢。
You can read kafka as ConsumerRecords<Long, String>
. 您可以将kafka读为
ConsumerRecords<Long, String>
。 And then SolidList.parseFrom(ByteBuffer.wrap(record.value().getBytes("UTF-8")));
然后是
SolidList.parseFrom(ByteBuffer.wrap(record.value().getBytes("UTF-8")));
Try implementing a Deserializer and pass it to KafkaConsumer constructor as value deserializer. 尝试实现反序列化器并将其作为值反序列化器传递给KafkaConsumer构造函数。 It could look like this:
它可能看起来像这样:
class SolidListDeserializer implements Deserializer<SolidList> {
public SolidList deserialize(final String topic, byte[] data) {
return SolidList.parseFrom(data);
}
...
}
...
KafkaConsumer<Long, SolidList> consumer = new KafkaConsumer<>(props, new LongDeserializer(), new SolidListDeserializer())
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.