Spring Boot中與Apache Kafka的Json消息通信

Question

我已經實現了兩種不同的應用程序，一種用於生產者，另一種用於消費者，並且在 apache kafka 的支持下傳遞了消息。 當我發布 String 消息時，通信正確完成，但是當我傳遞 Json 消息時，發生了以下錯誤。

錯誤

java.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer

Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition Kafka_Example_Json-0 at offset 0. If needed, please seek past the record to continue consumption.

Caused by: java.lang.IllegalArgumentException: The class 'com.benz.kafka.api.model.User' is not in the trusted packages: [java.util, java.lang, com.benz.kafka.consumer.api.model, com.benz.kafka.consumer.api.model.*]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (*).

ConsumerConfig 類

@Configuration
@EnableKafka
public class KafkaConfig {

private ConsumerFactory<String, User> userConsumerFactory()
    {
        Map<String,Object> config=new ConcurrentHashMap<>();

        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"127.0.0.1:9092");
        config.put(ConsumerConfig.GROUP_ID_CONFIG,"group_json");
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        config.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, JsonDeserializer.class);
        config.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS,JsonDeserializer.class.getClass());
        config.put(JsonDeserializer.TRUSTED_PACKAGES,"*");

          return new DefaultKafkaConsumerFactory<>(config,new StringDeserializer(),new JsonDeserializer<>(User.class));
    }

 @Bean
    public ConcurrentKafkaListenerContainerFactory<String,User> userKafkaListenerContainerFactory()
    {
        ConcurrentKafkaListenerContainerFactory<String,User> factory
                =new ConcurrentKafkaListenerContainerFactory<>();

        factory.setConsumerFactory(userConsumerFactory());

        return factory;

    }
}

模型

@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
public class User {

    private int userId;
    private String userName;
    private double salary;

    @Override
    public String toString() {
        return "User{" +
                "userId=" + userId +
                ", userName='" + userName + '\'' +
                ", salary=" + salary +
                '}';
    }
}

ProducerConfig 類

@Configuration
public class KafkaConfig {

    private ProducerFactory<String,User> producerFactory()
    {
        Map<String,Object> config=new ConcurrentHashMap<>();

          config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"127.0.0.1:9092");
          config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
          config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

          return new DefaultKafkaProducerFactory<>(config);
    }

    @Bean
    public KafkaTemplate<String,User> kafkaTemplate()
    {
        return new KafkaTemplate<>(producerFactory());
    }


}

模型

@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
public class User {

    private int userId;
    private String userName;
    private double salary;

    @Override
    public String toString() {
        return "User{" +
                "userId=" + userId +
                ", userName='" + userName + '\'' +
                ", salary=" + salary +
                '}';
    }
}

Answer 1

您需要使用@Bean 對其進行注釋並添加此配置：

    @Bean
    private ProducerFactory<String,User> producerFactory()
{
    Map<String,Object> config=new ConcurrentHashMap<>();

      config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"127.0.0.1:9092");
      config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
      config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

      return new DefaultKafkaProducerFactory<>(config);
}

@Bean
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        return props;
    }

    @Bean
    public ProducerFactory<String, AccountEvent> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }

    @Bean
    public KafkaTemplate<String, AccountEvent> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

如果你想使用帶有 kafka 基礎設施的項目，我在我的 github 中有它。 https://github.com/gabryellr/banking-system

Answer 2

這個

config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);

應該

config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);

但是，不清楚為什么您的受信任包配置沒有被應用； 建議您在JsonDeserializer.configure()方法中設置斷點。

編輯

哦...

return new DefaultKafkaConsumerFactory<>(config,new StringDeserializer(),new JsonDeserializer<>(User.class));

當您傳入這樣的反序列化器實例時，不會使用這些屬性； 您必須完全自己構建和配置解串器。

由於您無論如何都想使用ErrorHandlingDeserializer ，因此您應該在此處傳遞其中一個。

或者，將其更改為

return new DefaultKafkaConsumerFactory<>(config);

Answer 3

我已經找到了發生此錯誤的原因。 您可以看到在錯誤部分模型類下創建了兩個不同的包。 在Producer中，在com.benz.kafka.api.model包下創建模型類，在Consumer部分，在com.benz.kafka.consumer.api.model包下創建模型。 這是根本原因，我將com.benz.kafka.consumer.api.model更改為com.benz.kafka.api.model然后它起作用了。