简体   繁体   English

使用Spring Kafka反序列化来自同一Kafka主题的不同JSON有效负载

[英]Deserializing different JSON payload from same Kafka topic with Spring Kafka

I'm trying to deserialize different JSON payloads from the same Kafka topic. 我正在尝试反序列化来自同一Kafka主题的不同JSON负载。 The other questions asked here, guided me to a first attempt, but I was not able to get it running. 在这里提出的其他问题使我第一次尝试,但是我无法使其运行。

As Gary mentioned ( here ) there is some hint ( JsonSerializer.ADD_TYPE_INFO_HEADERS ), but when I send and receive both messages I get an exception. 正如Gary提到的( 这里 ),有一些提示( JsonSerializer.ADD_TYPE_INFO_HEADERS ),但是当我发送和接收这两个消息时,我都会遇到异常。

org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.foo.message.ConsumerImpl.consumeSelf(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.util.Map<java.lang.String, java.lang.Object>,com.foo.message.KafkaMessage,org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.String, java.lang.Object>)]
Bean [com.foo.message.ConsumerImpl@6df2a206]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [com.foo.message.KafkaMessageWithAdditionalField] to [com.foo.message.KafkaMessage] for GenericMessage [payload=com.foo.message.KafkaMessageWithAdditionalField@4e3168f7, headers={kafka_offset=22, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@c0e2fcf, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=fromBar, kafka_receivedTimestamp=1548310583481}], failedMessage=GenericMessage [payload=com.foo.message.KafkaMessageWithAdditionalField@4e3168f7, headers={kafka_offset=22, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@c0e2fcf, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=fromBar, kafka_receivedTimestamp=1548310583481}]
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:292) ~[spring-kafka-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:79) ~[spring-kafka-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:50) ~[spring-kafka-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1207) [spring-kafka-2.2.2.RELEASE.jar:2.2.2.RELEASE]

... ...

The LoggingErrorHandler mentions already a (the correct) value in ConsumerRecord. LoggingErrorHandler已经在ConsumerRecord中提到了一个(正确的)值。

2019-01-24 07:16:27.630 ERROR 27204 --- [ntainer#2-0-C-1] o.s.kafka.listener.LoggingErrorHandler   : Error while processing: ConsumerRecord(topic = fromBar, partition = 0, offset = 22, CreateTime = 1548310583481, serialized key size = -1, serialized value size = 196, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = com.foo.bar.message.KafkaMessageWithAdditionalField@4e3168f7)

First my config: 首先我的配置:

@EnableKafka
@Configuration
public class KafkaConsumerConfig {
    @Value("${spring.kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Bean
    public ConsumerFactory<String, KafkaMessage> consumerFactoryMessage()
    {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);           
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),
                new JsonDeserializer<>(KafkaMessage.class));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, KafkaMessage> kafkaListenerMessageContainerFactory()
    {
        ConcurrentKafkaListenerContainerFactory<String, KafkaMessage> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactoryMessage());
        return factory;
    }

    @Bean
    public ConsumerFactory<String, KafkaMessageWithAdditionalField> consumerFactoryMessageWithAdditionalField()
    {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),
                new JsonDeserializer<>(KafkaMessageWithAdditionalField.class));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, KafkaMessageWithAdditionalField> kafkaListenerMessageWithAdditionalFieldContainerFactory()
    {
        ConcurrentKafkaListenerContainerFactory<String, KafkaMessageWithAdditionalField> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactoryMessageWithAdditionalField());
        return factory;
    }
}

Here are the listeners: 这是听众:

    @KafkaListener(topicPartitions = @TopicPartition(partitions = "0", topic = "${foo.kafka.topic-springBoot}"), containerFactory = "kafkaListenerMessageContainerFactory")
    public void consumeSelf(@Headers Map<String, Object> map, KafkaMessage message, ConsumerRecord<String, Object> cr)
    {
        log.info("message received %s", message);
    }

    @KafkaListener(topicPartitions = @TopicPartition(partitions = "0", topic = "${foo.kafka.topic-springBoot}"), containerFactory = "kafkaListenerMessageWithAdditionalFieldContainerFactory")
    public void consumeSelfAdd(@Headers Map<String, Object> map, KafkaMessageWithAdditionalField message, ConsumerRecord<String, Object> cr)
    {
        log.info("messageKafkaMessageWithAdditionalField received %s", message);
    }

You can't do that; 你不能那样做; you have 2 different listener containers with listeners that expect different objects. 您有2个不同的侦听器容器,其中的侦听器期望使用不同的对象。

For multiple listener methods that receive different types, you need to use @KafkaListener at the class level and @KafkaHandler at the method level. 对于接收不同类型的多个监听器方法,你需要使用@KafkaListener类级别和@KafkaHandler在方法级别。

See @KafkaListener on a Class . 请参见有关类的@KafkaListener

When using @KafkaListener at the class-level, you specify @KafkaHandler at the method level. 在类级别使用@KafkaListener时,请在方法级别指定@KafkaHandler。 When messages are delivered, the converted message payload type is used to determine which method to call. 传递消息时,将使用转换后的消息有效负载类型来确定要调用的方法。

@KafkaListener(id = "multi", topics = "myTopic")
static class MultiListenerBean {

    @KafkaHandler
    public void listen(String foo) {
        ...
    }

    @KafkaHandler
    public void listen(Integer bar) {
        ...
    }

    @KafkaHandler(isDefault = true`)
    public void listenDefault(Object object) {
        ...
    }

}

The default method is optional and is used for unknown payload types. 默认方法是可选的,用于未知的有效负载类型。

But this only works with a smart deserializer (that knows how to convert to different payloads). 但这仅适用于智能解串器(知道如何转换为不同的有效负载)。

Or, you can add a RecordFilterStrategy to the listener container factory to skip the other records in each listener. 或者,您可以将RecordFilterStrategy添加到侦听器容器工厂,以跳过每个侦听器中的其他记录。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Spring Kafka:同一主题上的不同json负载 - spring kafka : different json payload on the same topic 使用 Spring Kafka 从单个主题反序列化不同的消息(使用共享数据) - Deserializing different messages (With shared data) from single topic with Spring Kafka 在 Spring 中将 json 负载发送到 Apache Kafka 主题 - Send a json payload to Apache Kafka topic in Spring Quarkus 从 Kafka 主题拉取并将 JSON 有效负载发送到 REST 端点 - Quarkus pulling from Kafka Topic and Sending JSON Payload to a REST endpoint 如何在 Spring Boot 中反序列化 Kafka 主题中的 Json 字符串 - How to deserialize Json string from Kafka topic in spring boot 使用来自同一 kafka 主题的字符串和 JSON 消息 - 反序列化问题 - Consume String and JSON message from same kafka topic - issue with deserialization 用 spring 管理 Kafka Topic - Managing Kafka Topic with spring ErrorChannel中的ErrorMessage的有效载荷不包含Kafka主题 - Payload of the ErrorMessage in ErrorChannel does not contain Kafka Topic Kafka &amp; Spring Batch - 如何仅读取来自同一主题的未提交消息? - Kafka & Spring Batch - How to read ONLY uncommitted messages from the same topic? Spring Kafka多个使用者针对单个主题消耗不同的消息 - Spring Kafka multiple consumer for single topic consume different messages
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM