Kafka - Confluent Hub - Exploit only part of it

Question

I already saw a similar question in SO, but not clearly answer my doubts.

We have different Kafka clusters and lot of exploitation operational habits around it. We have our way to start/stop the cluster, lots of exploit scripts that help maintain the cluster etc..

Now we would like to use Kafka connect connectors for new needs, but from what I saw, Kafka connect is extremely coupled to confluent-hub. It's like I can't even use the connectors without having to install a full operational confluent-hub.

This makes it very difficult for us to use Kafka connect connectors, I understand that confluent-hub might be a framework that help running those connectors, but it's like we can't even use a dissociated Kafka cluster ( a one not exploited by confluent-hub..).

But maybe I miss something..

Do you know if there is any way to use properly Kafka connectors on a already existing Kafka cluster ( completely independent from confluent-hub) ?

EDITED :

It's more a question regarding the high coupled behaviour between confluent-hub and Kafka-connect. All the features that comes with Kafka connect ( distributed workers to handle different fail over scenarios, etc..) are not usable without confluent-hub, thus a "need" to have Kafka cluster running exclusively via confluent-hub, which is not an easy task when you already have an existing big Kafka cluster with lots of OPS habits on it.

Answer 1

Kafka Connect is part of Apache Kafka . It's a pluggable framework for streaming integration between systems in and out of Kafka.

To use Kafka Connect you need connectors for the specific technology with which you want to integrate. For example, S3 sink , Elasticsearch sink , JDBC source or sink , and so on.

The connector API is part of Apache Kafka, and available for anyone who wants to develop a connector.

Connectors are written by various people and organisations, and available in various different ways. How you obtain a connector depends on which connector you want, how its licensed, and how the author has made it available for distribution. It could be you go to github, clone the repo and build the JAR. It could be you can download the JAR directly.

All that Confluent Hub does is make lots of these connectors available for you in one place, easily searchable, and with an optional CLI tool that will install them for you.

Do you have to use Confluent Hub? No, not at all . Might it make your life easier in locating connectors that you want to use, and make it easier to install them? Hopefully :)

Disclaimer: I work for Confluent.