在这个例子中使用 Kafka 有什么意义，为什么不直接使用 DB？

Question

Here is an example of how Kafka should run for a Social network site. But it is hard for me to understand the point of Kafka here. We would not want to store posts and likes in Kafka as they will be destroyed after some time. So kafka should be an intermediate storage between View and DB. But why would we need it? Wouldn't it be better to use DB straightaway.

I guess that we could use kafka as some kind of cache so the data accumulates in Kafka and then we can insert it to DB in one big batch query. But I am pretty sure that is not the reason kafka here.

Answer 1

What's not shown in the diagram is the processes querying the database (RocksDB, in this case). Without using Kafka Streams , you'd need to write some external service to run GROUP BY / SUM on the database.
With Kafka Streams Interactive Queries , that logic can be moved closer to the actual event source, and is performed in near real time, rather than a polling batch. In a streaming framework, you could also send out individual event hooks (websockets, for example) to dynamically update "likes per post ", "shares per post ", "trends", etc without needing the user to update the page, or have the page load AJAX calls with large API responses for those details for all page rendered items.

More specifically, each Kafka Stream instance serves a specific query, rather than the API hitting one database for all queries. Therefore, load is more distributed and fault tolerant.

Worth pointing out that Apache Pinot loaded from Kafka is more suited for such real time analytical queries than Kafka Streams.

Also as you pointed out, Kafka or any message queue would act as a buffer ahead of any database (not a cache, although, Redis could be added as a cache, just like the later mentioned search service). And there's nothing preventing you from adding another database that's connected to Kafka Connect sink. For instance, a popular design is to write data to a RDBMS as well as Elasticsearch for text based search-indexing. The producer code only cares about one Kafka topic, not every downstream system where the data is needed.