Spring Cloud Stream Kinesis 消费者组可以由非 Spring 生产者发送吗？

Question

In our environment, we have a producer that is not a Spring microservice sending Avro messages over Kinesis to a Spring Cloud microservice. My question is how can the non-Spring producer add the consumer group message header to each message so that the Spring microservice can scale up to more than one instance, ensuring that the Spring microservices do not consume each message more than once?

Can this be done in Kinesis, and if so, are there any code examples anyone can point me to?

Answer 1

I would say "Not clear what you're asking", but I'll try to explain what might be wrong.

The Consumer Group is an abstraction, well..., for consumers. The producer has nothing to know about that abstraction. It's point just send messages to the destination and forget.

One of main goals of the consumer group abstraction is to ensure that message is processed only in one consumer meanwhile there might be more than one consumer in a group.

There is a feature in AWS Kinesis complementing consumer group abstraction. It is sharding in Kinesis streams. With consumer group implementation in Spring Cloud Stream, there is a trick to distribute shards between consumers in a group, so they can be processed in parallel.

Here you really can benefit fro sharding saying to your producer to which one to send a record. See more info about partitionKey in Kinesis Producer API.