发送到 Kafka 主题时反序列化 object 时出错

Question

我是卡夫卡的新手。 我正在尝试向包含 header 和有效负载的 Kafka 主题发送消息。

以下是错误：

"org.apache.kafka.common.errors.SerializationException: Can't convert value of class com.cabservice.request.CabLocationPayload to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer\nCaused by: java.lang.ClassCastException: class com.cabservice.request.CabLocationPayload cannot be cast to class java.lang.String 

有效负载：{“标头”：{“eventName”：“CAB-LOCATION”，“eventId”：“3b1i333kiwoskl”，“时间戳”：1615205167470}，“有效负载”：{“cabId”：“cc8”，“driverId”： “test@gmail.com”，“geoLocation”：{“id”：“1234”，“纬度”：78.12，“经度”：45.23 } } }

我有 CabLocationPayload，其中包含字段 Header 和 Payload。

public class CabLocationPayload {

private Header header;

private Payload payload;

// getter 和 setter }

在 Controller 中，

@PostMapping(value = "/publish") public void sendMessageToKafkaTopic(@RequestBody CabLocationPayload cabLocationPayload) {

Header 和 Payload 具有 Json 的映射字段。

在 Producer 中更改 VALUE_SERIALIZER_CLASS_CONFIG 后，我可以看到数据。 但仍然因 ClassCastException 而失败。

{public class KafkaConfiguration { @Bean public ProducerFactory<String, String> producerFactoryString() { Map<String, Object> configProps = new HashMap<>();

    configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

    return new DefaultKafkaProducerFactory<>(configProps);
}

@Bean
public KafkaTemplate<String, String> kafkaTemplateString() {
    return new KafkaTemplate<>(producerFactoryString());
}

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> configProps = new HashMap<>();
    configProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    configProps.put(ConsumerConfig.GROUP_ID_CONFIG, "group_id");
    configProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    configProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);

    return new DefaultKafkaConsumerFactory<>(configProps);
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());

    return factory;
}

}}

当前错误是 {2 09:41:20.108 INFO 22561 --- [ad | producer-1] org.apache.kafka.clients.Metadata: [Producer clientId=producer-1] 集群 ID: lWghv-b_RG-_hO-qOp_cjA 2021-04-22 09:41:20.123 ERROR 22561 --- [nio- 9080-exec-2] oac.c.C.[.[.[/].[dispatcherServlet]: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.apache.kafka.common.errors.SerializationException: Can't convert value of class com.cabservice.request.CabLocationPayload to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer] with root原因

java.lang.ClassCastException: class com.cabservice.request.CabLocationPayload cannot be cast to class java.lang.String (com.cabservice.request.CabLocationPayload is in unnamed module of loader org.springframework.boot.devtools.restart.classloader. RestartClassLoader @1144043d; java.lang.String is in module java.base of loader 'bootstrap') at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:28) ~[kafka-clients-2.6. 0.jar:na]}

非常感谢任何帮助。

Answer 1

Kafka 配置“value.serializer”配置应该是 Serializer 子类，而不是您的 object 类型

例如

键：VALUE_SERIALIZER_CLASS_CONFIG，值：JsonSerializer.class（来源：org.springframework.kafka.support.serializer）

示例生产者配置：

    @EnableKafka
    @Configuration
    public class KafkaProducerConfiguration {
    
        @Bean
        KafkaTemplate<String, Object> kafkaTemplate() {
            return new KafkaTemplate<>(producerFactory());
        }
    
        @Bean
        public ProducerFactory<String, Object> producerFactory() {
            return new DefaultKafkaProducerFactory<>(getConfig());
        }
    
        private Map<String, Object> getConfig() {
            Map<String, Object> config = new HashMap<>();
    
            config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "brokers");
            config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
            config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
            return config;
        }
    
    }

示例消费者配置：

您必须将 Yourclass 替换为您想要使用的 class 名称。 （对于本例：CabLocationPayload）

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.support.serializer.ErrorHandlingDeserializer;
import org.springframework.kafka.support.serializer.JsonDeserializer;

import java.util.HashMap;
import java.util.Map;

@Configuration
public class KafkaConsumerConfiguration {


    private Map<String, Object> consumerConfigs() {
        final Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "your brokers");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "consumeer-group-id");
        return props;
    }

    @Bean
    public ConsumerFactory<String, YourClass> kafkaListenerConsumerFactory() {
        final ErrorHandlingDeserializer<YourClass> errorHandlingDeserializer = new ErrorHandlingDeserializer<>(new JsonDeserializer<>(YourClass.class, false));
        return new DefaultKafkaConsumerFactory<>(this.consumerConfigs(), new StringDeserializer(), errorHandlingDeserializer);
    }


    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, YourClass> kafkaListenerContainerFactory() {
        final ConcurrentKafkaListenerContainerFactory<String, YourClass> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(this.kafkaListenerConsumerFactory());
        return factory;
    }

}

Answer 2

将 Kafka 配置移至 application.properties

spring.kafka.consumer.bootstrap-servers: localhost:9092 spring.kafka.consumer.group-id: group-id spring.kafka.consumer.auto-offset-reset: earliest spring.kafka.consumer.key-deserializer: org .apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer spring.kafka.consumer.properties.spring.json.trusted.packages=*

spring.kafka.producer.bootstrap-servers: localhost:9092 spring.kafka.producer.key-serializer: org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer: org.springframework.kafka. support.serializer.JsonSerializer

不确定spring.kafka.consumer.properties.spring.json.trusted.packages=*是否有所作为。