Kafka Connect：单个连接器或每个表连接器的方法

Question

I have a database say test and we are having multiple Kafka Debezium Connectors on it. Each connector is associate with one table.

My question is in terms of memory usage, which is a better approach:

1. One connector per database OR
2. One connector per table

Answer 1

Considering all performance factors, it is always a recommended approach to have a single source connector (multiple instances to share the load), with replicator or configuration file per database instance (test1, test2, test3 etc), having multiple tables, so the data ingress would be 1 table -> 1 topic .

You can have a better view at Oracle Golden Gate implementation pattern for the same.

https://rmoff.net/2018/12/12/streaming-data-from-oracle-into-kafka/

Answer 2

I think it really depends on your use case. I don't think there is a general approach for all the usecases. For example, at my current job, we decided to have 4 connectors that stream changes from the same database, but, each of them is streaming from a subset of tables. The main reason is that we don't want to have a single point of failure where a bad record inside DB can break all our usecases that rely on CDC, hence, we divided the tables and assigned them to a connector. Note that it's not good to have a lot of replication slots on the database also. So it really depends on your usecase.