使用 Kafka 在两个微服务之间同步数据

Question

I am trying to use kakfa to sync data for two microservices A & B.

A stages some data for a group of employees in a company in a database table. Then end user will trigger an event from UI to A 's backend service where it will send kakfa message(s) to a topic B is subscribed to.

B then takes the data either from a message or from a staged table, validates it and persist to its own database table.

Questions I have are..

1. Employees can range from 10 to 1000s per company, and there could be multiple companies trying to sync the data at certain time of the year. So performance is a concern. What would be a good way to divide the load? meaning.. should I design the message to be at employee level? This would mean there could be thousands of messages although design wise it would be the simplest. Or should it be at a company level? or group of employees within a company? Microservice is not doing much processing and persisting to the table.. Would it able to handle the load? What would be the limiting factor?

2. Data we are handling is JSON stored in DB. Would it be better to have a staging table and look up from B using some sort of primary key in the message? or is having all the data within a message be fine? JSON is not that big per employee data, but if aggregated to a group of employee let's say 100s, it may be 10-100 Kilobytes. Are we buying much from looking up data from the table?

3. We need to be able to track the status/errors, so that end user is aware of any issues and perform action to correct data and/or try resync. Some approach I thought of was creating a table, call it BATCH_JOB and BATCH_TASK table to keep track of request at the job level (UI event for a group of employees as mentioned which trigger the resync process) and task (employee level). Or would there be a cleaner approach?

Any help/tips of design would be appreciated.

Answer 1

What would be a good way to divide the load?

The short answer is using custom partitioning schemes with a reasonably large number of partitions. Say 100.

Or you can create a topic per company, depending on if you are using different record schemas per topic

Are we buying much from looking up data from the table?

Well, you cannot query a topic as easily as a table, so that's the benefit... You could also use a KTable and interactive queries

Data we are handling is JSON stored in DB

I assume you're not just putting one BLOB column into the database (and you've not clarified what database you're using either).

Personally, I'd suggest you use Avro and Kafka Connect to sink topics into databases. That's the recommended solution for such a task within the Kafka APIs without introducing other projects like Spark or writing your own database code

We need to be able to track the status/errors, so that end user is aware of any issues and perform action to correct data

Tables could work, but if you can write records to a table, you can also write events to another Kafka topic and get "notifications" from that