[英]Spring Cloud Stream Kafka > consuming Avro messages from Confluent REST Proxy
我有以下場景:
我的應用程序如下所示:
@SpringBootApplication
@EnableBinding(Sink.class)
public class MyApplication {
private static Logger log = LoggerFactory.getLogger(MyApplication.class);
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
@StreamListener(Sink.INPUT)
public void myMessageSink(MyMessage message) {
log.info("Received new message: {}", message);
}
}
而MyMessage是Avro從Avro架構創建的類。
我的application.properties看起來像這樣:
spring.cloud.stream.bindings.input.destination=myTopic
spring.cloud.stream.bindings.input.group=${spring.application.name}
spring.cloud.stream.bindings.input.contentType=application/*+avro
我的問題是,每次收到新消息時,都會拋出以下異常:
org.springframework.messaging.MessagingException: Exception thrown while invoking MyApplication#myMessageSink[1 args]; nested exception is org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -27
at org.springframework.cloud.stream.binding.StreamListenerAnnotationBeanPostProcessor$StreamListenerMessageHandler.handleRequestMessage(StreamListenerAnnotationBeanPostProcessor.java:316) ~[spring-cloud-stream-1.1.0.RELEASE.jar:1.1.0.RELEASE]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
...
Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -27
at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:430) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readMapKey(GenericDatumReader.java:335) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:321) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:177) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:116) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144) ~[avro-1.8.1.jar:1.8.1]
at org.springframework.cloud.stream.schema.avro.AbstractAvroMessageConverter.convertFromInternal(AbstractAvroMessageConverter.java:91) ~[spring-cloud-stream-schema-1.1.0.RELEASE.jar:1.1.0.RELEASE]
at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:175) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.converter.CompositeMessageConverter.fromMessage(CompositeMessageConverter.java:67) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.annotation.support.PayloadArgumentResolver.resolveArgument(PayloadArgumentResolver.java:117) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:138) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.cloud.stream.binding.StreamListenerAnnotationBeanPostProcessor$StreamListenerMessageHandler.handleRequestMessage(StreamListenerAnnotationBeanPostProcessor.java:307) ~[spring-cloud-stream-1.1.0.RELEASE.jar:1.1.0.RELEASE]
... 35 common frames omitted
據我所知,問題在於Confluent堆棧包含消息模式的ID作為消息有效負載的一部分,並且客戶端應該在模式ID之后開始讀取實際的Avro消息。 我似乎需要配置Kafka綁定以使用Confluent的KafkaAvroDeserializer,但我無法弄清楚如何實現這一點。
(我可以使用Confluent的avro控制台消費者完全正確地檢索消息,因此它似乎不是Avro編碼的問題)
我也嘗試使用@EnableSchemaRegistry注釋並配置ConfluentSchemaRegistryClient bean,但它在我看來這樣只能控制存儲/檢索模式的位置,而不是實際的反序列化。
這甚至應該以某種方式起作用嗎?
在設置per-binding
屬性spring.cloud.stream.kafka.bindings.input.consumer.configuration.value.deserializer
以使Confluent's KafkaAvroDeserializer class name
是否有效?
回答我自己的問題。 我現在所做的是實現一個MessageConverter,它只是刪除任何消息的前4個字節,然后再將它們傳遞給Avro解碼器。 代碼主要來自spring-cloud-stream的AbstractAvroMessageConverter:
public class ConfluentAvroSchemaMessageConverter extends AvroSchemaMessageConverter {
public ConfluentAvroSchemaMessageConverter() {
super(new MimeType("application", "avro+confluent"));
}
@Override
protected Object convertFromInternal(Message<?> message, Class<?> targetClass, Object conversionHint) {
Object result = null;
try {
byte[] payload = (byte[]) message.getPayload();
// byte array to contain the message without the confluent header (first 4 bytes)
byte[] payloadWithoutConfluentHeader = new byte[payload.length - 4];
ByteBuffer buf = ByteBuffer.wrap(payload);
MimeType mimeType = getContentTypeResolver().resolve(message.getHeaders());
if (mimeType == null) {
if (conversionHint instanceof MimeType) {
mimeType = (MimeType) conversionHint;
}
else {
return null;
}
}
// read first 4 bytes and copy the rest to the new byte array
// see https://groups.google.com/forum/#!topic/confluent-platform/rI1WNPp8DJU
buf.getInt();
buf.get(payloadWithoutConfluentHeader);
Schema writerSchema = resolveWriterSchemaForDeserialization(mimeType);
Schema readerSchema = resolveReaderSchemaForDeserialization(targetClass);
DatumReader<Object> reader = getDatumReader((Class<Object>) targetClass, readerSchema, writerSchema);
Decoder decoder = DecoderFactory.get().binaryDecoder(payloadWithoutConfluentHeader, null);
result = reader.read(null, decoder);
}
catch (IOException e) {
throw new MessageConversionException(message, "Failed to read payload", e);
}
return result;
}
然后,我通過application.properties將傳入的Kafka主題的內容類型設置為application / avro + confluent。
這至少讓我檢索使用Confluent堆棧編碼的消息,但當然它不會以任何方式與架構注冊表交互。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.