Kafka连接接收器连接器与多个分区主题

Question

We would like to use the Kafka connect sink connector in order to copy messages from Kafka to Mongo DB. In our use case, we have multiple topics, with one partition each (the name of the topics can be regexed, for example, topic.XXX.name). The number of these topics is increasing continuously. I wonder if the Kafka connect architecture fits this use case. If so, how can it be configured the gain high scaleability and parallelism? What will be the tasks.max? Number of workers?

Answer 1

Kafka Connect is flexible; the answer is as many as you need.

The number of running tasks per connect worker is mostly only limited by the JVM heap size of each worker. Adding more workers will allow you to have more total active connectors.

For sink connectors, however, you can only have as many total tasks as total topic partitions being consumed.

One thing to worry about, though, is frequent consumer group rebalancing as you add more and more topics. For this reason, it would be recommended to create independent connectors for any critical data