Kafka：加入事件形成多个主题

Question

We are using Kafka in over a year now and want to move on with a deeper integration. But there is one concept I struggle with. I will try to explain what we want to achieve and how the solution looks like we came up with. The issue from my perspective, it's not really an elegant way and that's why I'm questioning if I got it right.

The Problem

We have one stream with complex structured events (nested structure). A consumer is taking those events, ripping them apart and is putting those pieces in separate topics. Behind each topic are a lot of different other service who enrich the events flowing through the topics in a stream processing fashion. At the end we have a number of topics, each with partial enriched events and we want to move them all together to bring them back as one complete event at the end of the entire process. But this easier said than done.

The Solution

At the end we have one service that consumes all the topics and builds a buffer until all the partial events flown in and puts them together to publish it in a new topic. The challenge for is to make sure we only produce events in the final topics that are complete. This works but has some pitfalls

• the buffer can not be internal, it has to be something external that multiple consumer can share informations
• we can theoretically run in timing issues and create dead entries
• we can have consistency issues
• and so on

My Question

Even if it works and I don't think this is a very elegant way, are we on the right track or did we misunderstood something in the concepts and handling of Kafka events and stream processing? Are there any better ways to do it? Has somebody experience with this and can share some leanings or ways to integrate it in stable way?

Thanks. Any comments are very appreciated.

Answer 1

We have gone through a very similar use case and architecture. I understand you are splitting the initial nested message into multiple topics to increase the parallelism and therefore increase the throughput.

In our experience, this leads to a highly complex architecture because joining streams (as you have already described) can be very hard to operate. The main issues we had were:

• If one of the enrichment jobs is failing, how long should all the other messages wait? It could take a few hours or a few days to fix the bug.
• If those enrichment jobs depend on external system and they are not reachable for some time. How long should you wait to make them available again?

In our experience the current approach you are describing brings in a lot of complexity and (sometimes uncontrollable) dependencies.

In the end we kept all the data together and significantly increase the partitions of the topic to increase the throughput. That way each message is consistent in itself and if there is a problem with any enrichment, the entire message is impacted and not just part of it. To reduce the complexity in a single job we have buffered interim data in Kafka topics, in your case this could mean to have the various enrichment job run in sequence and not in parallel with a topic in between them. That way each job stays reasonable small and you can make use of the re-play functionality that comes with Kafka.

Operating stream joins is one of the most sophisticated things to do and I recommend to avoid it as much as possible unless it is acceptable to have short term inconsistencies and you are not required to procces all messages but instead might drop the one or the other.