多个组共享相同配置的分布式 Kafka 源连接器

Question

I have two Kafka connector nodes say Node-A and Node-B.

I run one distributed worker (pointing to the same Kafka cluster) in each node.

The group id and client id are unique across both the distributed workers.

Connector tasks are created successfully using http POST requests on both the nodes. The connector tasks produce messages for the same topic.

Note: I understand that the distributed workers are meant to have same group id for distribution of tasks. But I am doing as mentioned above for a specific case as an experiment.

I use the http GET request from Node-A say, as in http://localhost:8083/connectors

The http response (from Node-A) displays the connectors created from both the nodes ie from Node-A and Node-B, whereas I expect that the response should yield only the connectors created from Node-A and not those created from Node-B.

I want to know what is shared among these two distributed workers and how can i override this sharing?

Answer 1

The http response (from Node-A) displays the connectors created from both the nodes ie from Node-A and Node-B

Sounds like both groups are sharing the same config and status topics.

status.storage.topic
offset.storage.topic
config.storage.topic

As per the documentation, it's recommended you read the description of what settings each topic should have, and create the topics manually

https://kafka.apache.org/documentation/#connect_running