Why is the Kafka distributed connector dying when the node I created it on is killed?

Question

I'm launching a Kafka connector in distributed mode in a local 'launch' Docker container (separate to the Kafka node container). The connector works as expected, but when I kill the launch container the connector stops working. I expected it to continue working since I believed it to be registered and running on a worker on the Kafka node in a different container. My setup in more detail follows:

Currently I'm running everything through Docker containers locally. I have:

1. A Zookeeper node (3.4.9)
2. A Kafka node (Apache, 0.10.1.0)
3. A 'launch' node.

the launch node downloads the appropriate Kafka version and unzips it's contents. It then builds the connector source, sets the classpath to include the necessary JARs, then executes the connector as such:

connect-distributed.sh config/connect-distributed.properties

The distributed properties file sets the group id, the various topic names, schemas and converters and also the bootstrap servers (which point to the Kafka node (2) above).

This command seems to execute properly, with the restful connector http service being started successfully. I can then issue the POST request to http://example:8083/connectors , supplying the configuration for the connector tasks. The command completes without error and the connector is successfully started. I can consume from a topic in the Kafka node (2) and I see output that indicates the connector is working and sending data through.

When I kill the launch node (3) I expect the connector to continue running since I registered it with the Kafka cluster, albeit a cluster of one. The connector does not continue to run and appears to die with the launch node. Isn't the connector supposed to be managed now by a worker in the cluster? Do I need to change how I'm launching the connector or am I misunderstanding something?

Answer 1

Kafka Connectors do not execute on the Kafka brokers. They are executed in "Kafka Connect Worker" processes, which is what your question is calling "a 'launch' node". These processes accept REST requests for connectors and run the connectors in the worker processes . Under the hood, those processes are simply interacting with the Kafka brokers via normal producers and consumers. Kafka Connect is providing a framework on top of those clients to make it easy to build scalable connectors so connector developers only need to focus on how to pull or push data to the system the connector is written for. This means that processing only continues if at least one worker process is still alive.

There are two types of worker processes. In standalone mode, the connector configuration is not persisted anywhere -- you generally pass it in via the command line. Offset information (ie which data you've already copied) is maintained on the local filesystem. Therefore, in this mode, you can only assume you'll resume where you left off if you restart the process on the same node with access to the same filesystem.

In distributed mode, the workers coordinate to distribute the work and they share common, persistent storage (in Kafka) for connector configs, offsets, etc. This means that if you start up one instance and create a connector, shutting down that instance will halt all work. However, when you start an instance again, it will resume where it left off without re-submitting the connector configuration because that information has been persisted to Kafka. If you start multiple instances, they will coordinate to load balance the tasks between them, and if one instance fails (due to crash, elastically scaling down the # of instances you are running, power failure, etc), the remaining instances will redistribute the work automatically.

You can find more details about workers, the different types, and how failover works in distributed mode in Confluent's Kafka Connect documentation