multiple connectors in kafka to different topics are going to same node

Question

I have created two kafka connectors in kafka-connect which use the same Connector class but have different topics they listen to.

When I launch the process on my node, both the connectors end up creating tasks on this process. However, I would like one node to only handle one connector/topic. How can I limit a topic/connector to a single node? I don't see any configuration in connect-distributed.properties where a process could specify which connector to use.

Thanks

Answer 1

Kafka Connect in distributed mode can run as a cluster of one or more workers . Each worker can run multiple tasks . Depending on how many connectors and workers you are running, you will have tasks running on the same worker . This is deliberate - the idea is that Kafka Connect will manage your tasks and workload for you, across the available workers.

If you want to isolate your processing you can run Kafka Connect as separate Connect clusters, either on the same machine (make sure to use different REST ports), or separate machines.

For more info, see architecture and config for steps to configure separate clusters. Note that a cluster can actually be a single worker, but then you don't have any redundancy in the event of failure.