微服务与卡夫卡消费者集团

Question

We are working on a appliation in which we are using kafka.

The components of the Application are as follows,

1. We have a microservice which gets the request's and pushes the messages to a kafka topic. (Lets say ServiceA)

2. Another microservice which consumes the messages from topics and push the data to a datastore. (Lets say ServiceB)

I am clear with ServiceA part of the application but have some design confusions in the ServiceB part.

As ServiceB we are planning for REST API,

1. Is it good to bundle Consumer and controllers in a single application ?
2. For consumer i am planning to go with ConsumerGroup with multiple Consumer's to acheive more throughput. Is there any better and efficent approach ?
3. Should i take out the Consumer part of ServiceB and make it as a separate service which is independent ?
4. If we are bundling it inside the ServiceB should i configure Consumer as a Listener ? (We are going with spring boot for microservice)

Thanks in Advance !

Answer 1

Is it good to bundle Consumer and controllers in a single application ?

Its good to bundle together by context, having a listener, wich forwards to another service to controll makes no sense in my opionion. But consider splitting up controller by different context if necessary. Like Martin Fowler says: start with a monolith first and than split up ( https://martinfowler.com/bliki/MonolithFirst.html )

For consumer i am planning to go with ConsumerGroup with multiple Consumer's to acheive more throughput. Is there any better and efficent approach ?

A consumer group makes sense if you think about scale your service B out. If you want to have this possibility in the future, start with one instance of ServiceB inside the consumer group. If you use something like kubernetes, its simple do later on deploy more instances of your service if required. But do not invest to much on in an eventual future. Start simple and do some monitoring, and if you figure out some bottle necks, than act. One more thing to keep in mind, is that kafka by default keeps message for a long time (i guess 7 days by default) so if you thing in a classical message broker style you could get a lot of duplicates of your messages. Think about a update message, if somethings change, which is raised when your ServiceA starts. Maybe reducing the retention.ms would be an option, but take care not to loose messages.

Should i take out the Consumer part of ServiceB and make it as a separate service which is independent ?

No i think not.

If we are bundling it inside the ServiceB should i configure Consumer as a Listener ? (We are going with spring boot for microservice)

Yes :-)