如何关联来自多个主题的 kafka 事件？

Question

I have a springboot application that consume from topic A and from topic B. Events from topic A are used to create entries in the db and events from topic B use this information to perform a task. As one suspects, events from topic B relay on events from topic A arriving first & creating entries in a db. So far I would just get events from topic A first and then get events from topic B after all the events from topic A were processed. So my events will come in the following order:

eventA1
eventA2
eventA3
eventB1 (would look for entry created by A1)
eventB2 (would look for entry created by A2)
eventB3 (would look for entry created by A3)

However, say I cannot relay on topic B sending events after topic A. In that case I can face a situation where events from topic B arrive first (and cannot do anything since events from topic A will come afterwards) as so:

eventB2 (would look for entry created by A2 but it has not been received yet)
eventB3 (would look for entry created by A3 but it has not been received yet)
eventA1
eventA2
eventB1 (would look for entry created by A1)
eventA3

So far my solution is to store events from topic B somewhere and use scheduler to periodically check whether the events from topic A arrived yet, so that events from topic B can do their work. That seems very not-ideal (and while I cannot put it in words, my intuition tells me that I can encounter a number of problems with this approach) and adds a fair share of complexity.

Something similar has been asked here and it seems that the answer is - just don't. Combine messages into one and don't mess around with joining streams. While I wish I could do that, unfortunately I can't.

Another interesting idea mentioned here is to feed topic A into topic B and then consume the topic containing both pieces. But it something I again cannot do unfortunately.

So my question is as follows: how can I make it so that if event from topic B arrived, and there is no event from topic A, it will wait until the relevant event from topic A is received? Is this even possible without needing to temp store events from topic B somewhere and create a scheduler? Is it possible to somehow tell kafka topics to wait for each other (events from both topics have a field that can be used to figure out which 2 event need each other)?

Answer 1

You can use the non-blocking retries feature that Spring for Apache Kafka provides:

https://docs.spring.io/spring-kafka/docs/current/reference/html/#retry-topic

If a B arrives too soon, throw an exception and it will move to the next topic in the retry sequence and won't be delivered again until the delay expires.

It is similar to what you are doing now, but all taken care of by the framework.