在Kafka中启动多个连接器通过单个分布式工作者连接吗？

Question

How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?

Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).

Currently, I am adding one connector at a time using following curl command.

curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'

Issue:

• For each new connector I add, previous/existing connector(s) restarts along with new connector.

Question:

• How should I start/create all these new connectors with one REST call in a distributed worker mode?
• Is there any way to have all connector configs in a single REST call, like an array of connector configs?

I tried to search for the same but didn't come across any workaround for this.

Thanks.

Answer 1

For each new connector I add, previous/existing connector(s) restarts along with new connector.

Yes, that's the current behaviour of Kafka Connect. For further discussion see:

• https://issues.apache.org/jira/browse/KAFKA-5505
• https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies

How should I start/create all these new connectors with one REST call in a distributed worker mode? Is there any way to have all connector configs in a single REST call, like an array of connector configs?

You can't do it in a single REST call

---

If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.

So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.

Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.