Using KAFKA as Hot storage and MySQL as Cold Storage

Question

I want to insert around 500k datas per second into KAFKA, and then synchronize them with MySQL database that support only 40k INSERTs/UPDATEs per second.

So in order to acheive my goal, i want use KAFKA as a hot storage since i need to use the datas as fast as possible and share them with multiple services and make updates quickly since it can't be done using MySQL alone (That i want to use as a cold storage to store the datas as long as possible and archive the changes).

Here a schema about my need: https://i.imgur.com/NyRn8zi.jpg

Is there any idea on how to process this case ? Or another way to acheive the same goal ?

I don't need a code but a ressources and path to follow (My main language is PHP).

Answer 1

Kafka connect JDBC acts as both a source and a sink. Therefore it can at least perform inserts, and updates might depend upon your Kafka message structure / primary keys in the table