Install Kafka connector manually

Question

I'm trying install twitter source connector and found the following links link1 andlink2 . The first one shows how to install twitter source connector and the second generally how to install manually a connector (always Confluent Platform).

However I'm confused from the entire procedure. I followed the steps in here , and so far it's working fine. However since the twitter source connector is not in the list of the Connector Hub it needs to be installed manually. According to link1, I clone a git repo somewhere - Question : where do I clone it? In the docker-compose.yml there are a bunch of services. In particular, a docker-compose ps after I spin up the yaml shows

       Name                    Command                  State                         Ports
  ------------------------------------------------------------------------------------------------------------

   broker            /etc/confluent/docker/run        Up             0.0.0.0:9092->9092/tcp
   connect           /etc/confluent/docker/run        Up             0.0.0.0:8083->8083/tcp, 9092/tcp
   control-center    /etc/confluent/docker/run        Up             0.0.0.0:9021->9021/tcp
   ksql-datagen      bash -c echo Waiting for K ...   Up
   ksqldb-cli        /bin/sh                          Up
   ksqldb-server     /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8088->8088/tcp
   rest-proxy        /etc/confluent/docker/run        Up             0.0.0.0:8082->8082/tcp
   schema-registry   /etc/confluent/docker/run        Up             0.0.0.0:8081->8081/tcp
   zookeeper         /etc/confluent/docker/run        Up             0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

I don't understand to which one of the above I should carry out the instructions of link1. Is it in the zookeeper, the broker or in the connect? Any help is welcomed, please if someone has time to write out a few words around the steps in link1 more thoroughly, please feel free. Cheers

Answer 1

Connectors are installed in Kafka Connect workers. They must be installed before the worker is started.

There is a newer version of that blog article, here: https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb . Associated with that is a new demo, which you can find here: https://github.com/confluentinc/demo-scene/tree/master/ksqldb-twitter

As part of the demo, it installs the twitter connector at runtime . It does this by overriding the default command of the container image and instead runs the connector installation before running the connect worker:

    command: 
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt jcustenborder/kafka-connect-twitter:0.3.33
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run & 
        #
        sleep infinity