Apache Kafka for inter-microservice communication; at what level to map topics to events?

Question

I'm working with a medium-sized web application, which is separated into several microservices. Currently, the way the services communicate with each other is an unsustainable mess, so I'm looking into other solutions.

One approach that looks very appealing to me is to use a message broker, where each service sends and listens to messages. Apache Kafka has caught my attention and looks like a very promising choice of software for this purpose.

However, I'm not sure how I'd use Kafka's topics, or more specifically, at what level to map the events to topics. I've identified three major levels of granularity, described below.

For demonstration purposes, consider a hypothetical online store consisting of a number of services such as ShoppingCart, Billing and Shipping.

The whole application uses one topic .

One single topic, eg my-app-events , is used to channel all events sent and received by the services. An event might look like: {"from": "shopping-cart", "name": "PRODUCT_ADDED", "payload": {"product_id": 137}} .

Each microservice uses its own topic .

Separate topics, such as shopping-cart-events , billing-events , shipping-events etc. are used. Now an event in the shopping-cart-events topic might look like {"name": "PRODUCT_ADDED", "payload": {"product_id": 137}} .

Each event type uses its own topic .

Here each possible message has its own topic. I suppose it'd make sense to include the producer of the event in the topic name, so that a topic might be called shopping-cart.product-added . In that case, the message contents would simply be the payload, eg {"product_id": 137} .

---

I hope I made the difference clear between the three approaches. What do you suggest? Have you used Kafka (or any other message broker) successfully in this way? What are the advantages and pitfalls with each of the solutions?

Answer 1

The whole application uses one topic.

I really see little benefit to this

Each microservice uses its own topic.

If you need ordering between events that happened to the same entity, this is the way to go. eg events such as shopping-cart.product-added, shopping-cart.product-removed for the same shopping-cart ID for consistency should preserve ordering. That implies they go to the same partition, hence the same topic.

Each event type uses its own topic.

The benefit of this approach is type-safety, since you only get one type of messages in each topic, the deserialization and downstream handling is less error-prone. However, you can't preserve ordering between different events happening to the same entity.

All in all, I'd suggest one topic per entity type (entity being something that has the events happen to, in DDD terms this would be called an aggregate instead), eg shopping-cart. If your services are so granular that you only have one entity type per service, this boils down to one topic per service.

Answer 2

About Kafka:

First of all, consider if you really want to run Kafka as your message broker.

Kafka is fast, but there are a lot of subtleties behind it that make it harder to use than you might expect. If you are running only web services maybe other pub/sub technologies might be more appropriate.

Kafka is a topic on its own, so I will make it short on what you should regard more closely. This is opinionated, and from my own experience with the techno:

• you cannot easily read, browse or delete messages in your topics, good luck finding that one single buggy message if you do not perform proper logging
• the partitioning system requires extra-administration efforts because they use a different server API than the one used for topics by common libraries. If you run only a few consumers you might be fine with a single partition per topic.
• the offset commit feature: it is of big importance when you want to ensure that not a single message is missed, so you might want to disable autocommit and perform manual validation of offsets
• unless you guarantee idempotence, you will have to implement an 'exactly-once' feature by yourself, because Kafka is 'at least once': you do not want to debit that customer twice, or ship your products again, or spam-flood the customers because you mishandled Kafka's replayability features.
• kafka consumers classes usually 'block' to read until they have events, which might not be ideal if you run single-threaded processes and require to monitor other stuff... You might end up doing polls() but those implementations may vary according to your client library.
• the best monitoring tool out there for Kafka was made by Yahoo and is community-maintained. The official Kafka administration and monitoring tools are poor and laughable at.
• Kafka is still immature and I would not say it is production-safe but since 0.10 things are better.
• I don't know right now, but, months ago, most of the client JS libraries were either obsolete (Kafka <= 0.8), badly documented or painful to use.

About topic architecture:

First of all, @Michal Borowiecki gave some good points which are worth considering.

From my own experience, you will find convenient to have a topic per event type: whenever you open the topic you know what you will find in it.

If you need to consume different event types, you can do it by consuming multiple topics at the same time (beware, Kafka consumers are not really good at balancing, if you consume 5 topics and one is flooded you might get stuck into consuming all that is incoming from there without being given any data from other topics until things calm down... Reactivity may suffer).

One topic per event type should not prevent you to have different event names for a single event type, and it is fine if your service drops/filters events.

Eg: you can have an event type 'User Connexion' and have 'Logged in' and 'Logged out' as event names for that event type, all within the same topic (user-connexion) - the event name is embedded into the metadata of your event. This make sense because you want to ensure that those messages are handled in order: log-ins must precede log-outs and a logged out user cannot be allowed to perform privileged operations.

If you want to have a more global overview on how events are correlated, or replay events based on a cluster-wide order, you will probably have to implement correlation IDS , vector clocks or interval-tree clocks and proper storage of your messages for easier manipulation and inspection (eg: dumping the topics onto MongoDB...).

Also, if you use MongoDB, take a look at the findAndModify() feature of MongoDB which will allow you to consistently flag each message with a unique cluster-wide ID.