在Kafka Streams作业中执行数据库同步查询或静态调用是一种好习惯吗？

Question

I use Kafka streams to process real-time data, in the Kafka streams tasks, I need to access MySQL to query data, and need to call another restful service.

All the operations are synchronous.

I'm afraid the sync call will reduce the process capability of the streams tasks.

Is this a good practice? or Is there any good idea to do this?

Answer 1

A better way to do it would be to stream your MySQL table(s) into Kafka, and access the data there. This has the advantage of decoupling your streams app from the MySQL database. If you moved away from MySQL in the future, so long as the data were still written to the Kafka topic from wherever it subsequently lived, your streams app would be unaffected. If it's just configurations you're storing in MySQL, you could even adopt the pattern that some people use of using Kafka as the primary store for data (using log compaction, to retain it forever).