如何將連接器安裝到 apache 卡夫卡連接的 docker 圖像

Question

I am using https://hub.docker.com/r/1ambda/kafka-connect/ to install docker kafka connect on ubuntu.I am able to run it but,I am unable to install any more connectors in it.

我試過的

1-我嘗試將連接器文件從我的機器復制到 docker 容器連接器文件夾並重新啟動，但 api,http://localhost:8080/connectors 給出空數組。

2-還在 connect-distributed.properties 中添加了插件路徑。 如何做到這一點任何想法？

Answer 1

該圖像似乎沒有更新為與plugin.path一起使用，因為它使用的是 Kafka 0.10

您可以使用confluent-hub安裝連接器，就像我在 Connect 容器中記錄的那樣

https://github.com/OneCricketeer/apache-kafka-connect-docker

Answer 2

不使用 Confluent Hub 的解決方案

第 1 步：構建您自己的 Kafka-Connect 映像

您的目錄應如下所示：

my-directory /
  Dockerfile
  plugins /
    my-connector /
       <connector-jars>

Dockerfile：

FROM confluentinc/cp-kafka-connect-base:latest
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
ENV CONNECT_PLUGIN_PATH="/opt/kafka/plugins"
USER 1001

運行以下

cd /my-directory
sudo docker build . -t my-connector-image-name

第 2 步：為您的 kafka-connect 容器使用創建的映像

version: '3'
services:
        zookeeper:
                container_name: 'zookeeper'
                image: 'bitnami/zookeeper:latest'
                ports:
                        - '2181:2181'
                environment:
                        - ALLOW_ANONYMOUS_LOGIN=yes
        kafka:
                image: 'bitnami/kafka:latest'
                container_name: 'kafka'
                ports:
                        - '9092:9092'
                environment:
                        - KAFKA_BROKER_ID=1
                        - KAFKA_LISTENERS=PLAINTEXT://:9092
                        - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.178.70:9092
                        - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
                        - ALLOW_PLAINTEXT_LISTENER=yes
                depends_on:
                        - zookeeper
        kafka-connect:
                image: 'my-connector-image-name:latest'
                container_name: 'kafka-connect'
                ports:
                        - '8083:8083'
                environment:
                        - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
                        - CONNECT_REST_PORT=8083
                        - CONNECT_GROUP_ID=quickstart
                        - CONNECT_CONFIG_STORAGE_TOPIC=quickstart-config
                        - CONNECT_OFFSET_STORAGE_TOPIC=quickstart-offsets
                        - CONNECT_STATUS_STORAGE_TOPIC=quickstart-status
                        - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_REST_ADVERTISED_HOST_NAME=localhost
                depends_on:
                        - kafka

步驟 3 獲取可用的連接器插件

curl localhost:8083/connector-plugins | json_pp

關於我的 Kafka 配置的旁注：我將 Docker 托管在 Linux 虛擬機中，我分配了 IP 192.168.178。

Answer 3

1-我嘗試將連接器文件從我的機器復制到 docker 容器連接器文件夾並重新啟動，但 api,http://localhost:8080/connectors 給出空數組。

localhost:8083/connectors，將為您提供活動連接器的列表。 如果您需要檢查可用插件列表，您應該點擊 localhost:8083/connector-plugins

curl localhost:8083/connector-plugins
[{"class":"io.confluent.connect.activemq.ActiveMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.ibm.mq.IbmMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"5.5.1"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jms.JmsSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.s3.S3SinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.storage.tools.SchemaSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"io.confluent.kafka.connect.datagen.DatagenConnector","type":"source","version":"null"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"5.5.1-ccs"}, 
{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]

此外，需要將一個新連接器添加到所有工作節點的上述插件路徑中，並且所有工作節點都需要重新啟動

要進一步排除故障，您可以嘗試將日志級別增加到 DEBUG 並檢查啟動日志是否正在遍歷連接器路徑。您將看到具有以下格式的日志：

DEBUG Loading plugin urls: ......

希望這有助於解決您的問題:)