Kafka 动态间接寻址，带有类似于消息传递的发布/订阅

Question

I am new to Kafka, but have already a quite challenging problem to solve.

Before describing the problem: My application is about spatial indexing and geographic data coordination. Meaning, I really need the described type of re-routing for good reasons.

I need to achieve the following event flow:

• there are n instances of my application, and a large variable amount of data objects in total globally.
• each instance knows only about a subset of global objects. But there can be multiple instances knowing about the same data object.
• However, in case a global object is changed (including ones that are unknown at this point) the change must be propagated to all instances which know this data object. In that sense, „instances“ are subscribers to certain data objects.

Question 1: Is it wise to use Kafka with log compaction enabled to maintain a strongly consistent list of subscribers to an object? Eg:

• A topic named changeevents where all instances may publish changed data as required for this edge case.
• A topic named pubsub with a map from object id to subscriber topic:

ObjectId:[subscriberId1Tooic, subscriberId2Topic]

Question 2: What choices do I have in Kafka to make this re-routing work in the most scalable and low latency way? Is it possible to create dynamic routing events in-place, eg having a stream of change events and Kafka places the change event to all subscriber topics?

Question 3: This seems all a bit complicated. My scenario is pretty unique, yet I hope I'm missing something to make this less complicated?

It would be a valid question to ask at this point why I chose to use Kafka for what looks like a publish/subscribe problem. First, the data flow between the backend instances is in the common case not requiring this type of re-routing and the problem above is solving a < 1% case of total data to be processed. Secondly, I'm also investigating Apache Pulsar which seems to have better support for publish/subscribe scenarios. Where I struggle with is that my application is deployed by customers, and Pulsar has a far lower adoption / acceptance rate.

I researched about routing options in Kafka, and the closest I could find to this problem seems dynamic routing as described here https://www.confluent.io/blog/putting-events-in-their-place-with-dynamic-routing/

As I see it, I need to have an additional data source to maintain the pub/sub list, additional custom processors that place the messages to related subscriber topics, also all at the cost of duplicating messages.

Answer 1

Sure, a compacted topic seems like a reasonable approach if you want eventually consistent, unique data. You'll need a GlobalKTable like the linked post says to query against that data, though.

"Topics" themselves do not subscribe. You'd need a consumer that reads all "changeevents", then filters/branches into the downstream "client topics". This will likely be the largest bottleneck, and main way to scale it would be lots of partitions and record keys that map somehow to geographic regions, or some other unique identification.

It's unclear what your output data looks like; if you're publishing notifications or displaying some kind of map, you'll need another system listening to only "local" geo-fenced events. If you need to read those events multiple times, you'd probably need more than Kafka, such as Elasticsearch geospatial queries, if not another system that supports geopoint data (I recall there being a GIS plugin for Postgres). Eg Use Kafka Connect to write into that system, and anything that needs those events, just queries the database.