Kafka 和具有多种消息类型的主题（Avro）：Consumer（spring cloud spring 由于类路径上缺少 class 而失败

Question

Topic contains two types of messages: PaymentStarted and PaymentCompleted. There are 2 separated microservices with consumers. So:Microservice_1, has consumer1 which should grab the PaymentStarted type; Microservice_2, has consumer2 which should grab the PaymentCompleted type. Additionally Microservice_1 contains only PaymentStarted on its jvm classpath and Microservice_2 contains only PaymentCompleted on its jvm classpath.

Each service for handling different types of messages. I am using spring-cloud-stream, so in my Microservice_1 the Consumer 1 has:

@StreamListener(target = PayBindings.EVENTS, condition = "headers['eventType']=='com.company.domain.PaymentStarted'")
    void onPaymentEvents(Message<PaymentStarted> message){
    }

analogically Microservice_2 the Consumer 2 has:

@StreamListener(target = PayBindings.EVENTS, condition = "headers['eventType']=='com.company.domain.PaymentCompleted'")
    void onPaymentEvents(Message<PaymentCompleted> message){
    }

Documentations says( https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.0.M1/spring-cloud-stream.html#_using_streamlistener_annotation ) that filtering goes before the serialization.

But in my case before condition filters them out, the default deserialiser(in my case io.confluent.kafka.serializers.KafkaAvroDeserializer) fails due to lack of PaymentCompleted on the classpath at microservice_1. Analogically with microservice_2 and PaymentStarted. I don't want to mix the domains and keep PaymentStarted and PaymentCompleted pojos in Microservice_1 and Microservice_2.

I tried to solve that with my custom deserializer and it looks that it is doable with handling exception, but it is really tricky.

I am also confused that the doc I mentioned here above says that the filtering out goes before deserializing.

I appreciate your thoughts/comments.

Answer 1

You should really use a different topic for each type.

However, see Spring for Apache Kafka's ErrorHandlingDeserializer2 .

Which will catch the exception and the listener container will route the error directly to the container's error handler.

I am also confused that the doc I mentioned here above says that the filtering out goes before deserializing.

That does not apply when you are using native deserialization, which is much lower down in the stack.

Answer 2

You should consume all messages and inside the function onPaymentEvents filter messages needed be each consumer. I suppose you have a single microservice that producer both type of messages.Also I replace the Avro with JSON.