Kafka connect 找不到我的自定義寫入策略

Question

我正在嘗試根據此處的文檔為寫入 mongodb 的接收器連接器實施自定義寫入策略：

https://www.mongodb.com/docs/kafka-connector/current/sink-connector/fundamentals/write-strategies/

我試圖讓我的連接器將以下（虛擬）自定義連接器識別為概念驗證：

package com.fu.connect.sink;

import org.bson.*;

import com.mongodb.client.model.UpdateOneModel;
import com.mongodb.client.model.UpdateOptions;
import com.mongodb.client.model.WriteModel;
import com.mongodb.kafka.connect.sink.converter.SinkDocument;
import com.mongodb.kafka.connect.sink.writemodel.strategy.WriteModelStrategy;

import org.apache.kafka.connect.errors.DataException;

public class CustomWriteModelStrategy implements WriteModelStrategy {

    private static final UpdateOptions UPDATE_OPTIONS = new UpdateOptions().upsert(true);

    //incomming json should have one message key e.g. { "message": "Hello World"}
    @Override
    public WriteModel<BsonDocument> createWriteModel(SinkDocument document) {
        
        // Retrieve the value part of the SinkDocument
        BsonDocument vd = document.getValueDoc().orElseThrow(
                () -> new DataException("Error: cannot build the WriteModel since the value document was missing unexpectedly"));

        // extract message from incoming document
        BsonString message = new BsonString("");
        if (vd.containsKey("message")) {
            message = vd.get("message").asString();
        }

        // Define the filter part of the update statement
        BsonDocument filters = new BsonDocument("counter", new BsonDocument("$lt", new BsonInt32(10)));

        // Define the update part of the update statement
        BsonDocument updateStatement = new BsonDocument();
        updateStatement.append("$inc", new BsonDocument("counter", new BsonInt32(1)));
        updateStatement.append("$push", new BsonDocument("messages", new BsonDocument("message", message)));

        // Return the full update å
        return new UpdateOneModel<BsonDocument>(
                filters,
                updateStatement,
                UPDATE_OPTIONS
        );
    }
}

（從https://github.com/felixreichenbach/kafkaWriteStrategy/blob/master/src/main/java/de/demo/kafka/CustomWriteModelStrategy.java借來的）我正在編譯這個類和一些其他自定義轉換到 a.jar 和使用以下 Dockerfile 將它添加到我的插件路徑中：

FROM maven:3.6.0-jdk-11-slim AS build
COPY resources/custom_plugins /app/resources/custom_plugins
COPY resources/whitelist.csv /app/config/

WORKDIR /app/resources/custom_plugins
RUN mvn -e clean package

FROM confluentinc/cp-kafka-connect:7.2.2

ARG version
ENV VERSION=$version

USER appuser
RUN mkdir -p app/bin && \
    mkdir -p app/config
COPY --chown=appuser resources/truststore.jks app/config/
COPY --chown=appuser resources/whitelist.csv /app/config/

USER root

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-avro-converter:5.5.3 && \
    confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.8.0 && \
    mkdir /usr/share/confluent-hub-components/plugins && \
    mkdir /usr/share/confluent-hub-components/mongo_plugins && \
    cp /usr/share/confluent-hub-components/mongodb-kafka-connect-mongodb/lib/*.jar /usr/share/confluent-hub-components/mongo_plugins && \
    cp /usr/share/confluent-hub-components/confluentinc-kafka-connect-avro-converter/lib/*.jar /usr/share/confluent-hub-components/plugins && \
    cp /usr/share/filestream-connectors/*.jar /usr/share/confluent-hub-components/plugins

USER appuser

ENV ARTIFACT_ID=CustomPlugins-1.0-SNAPSHOT.jar
COPY --from=build /app/resources/custom_plugins/target/$ARTIFACT_ID /usr/share/confluent-hub-components/mongo_plugins/$ARTIFACT_ID

COPY --chown=appuser scripts/*.sh app/bin/
COPY --chown=appuser config/* app/config/

我當前的插件路徑：

plugin.path=/usr/share/confluent-hub-components/plugins,/usr/share/confluent-hub-components/mongo_plugins/CustomPlugins-1.0-SNAPSHOT.jar,/usr/share/confluent-hub-components/mongo_plugins/mongo-kafka-connect-1.8.0-confluent.jar,

在我設置的水槽屬性中

writemodel.strategy=com.fu.connect.sink.CustomWriteModelStrategy 

我嘗試了多種不同的路徑配置，包括但不限於將all.jars添加到同一目錄並只指定一個插件路徑，為SMT創建separate.jars和自定義寫入策略。 當我嘗試運行我的連接器時，我總是會遇到同樣的錯誤：

java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
Invalid value com.fu.connect.sink.CustomWriteModelStrategy for configuration writemodel.strategy: Class not found: com.fu.connect.sink.CustomWriteModelStrategy

我的自定義轉換工作正常，但據我所知，加載這些轉換的模塊與加載自定義寫入策略的模塊不同。

我已經嘗試在我能想到的所有排列中重構 Java 代碼和插件路徑，但我總是遇到同樣的錯誤

Answer 1

想通了：問題是由於這里發生的所有.jar文件復制造成的：

mkdir /usr/share/confluent-hub-components/plugins && \
mkdir /usr/share/confluent-hub-components/mongo_plugins && \
cp /usr/share/confluent-hub-components/mongodb-kafka-connect-mongodb/lib/*.jar /usr/share/confluent-hub-components/mongo_plugins && \
cp /usr/share/confluent-hub-components/confluentinc-kafka-connect-avro-converter/lib/*.jar /usr/share/confluent-hub-components/plugins && \
cp /usr/share/filestream-connectors/*.jar /usr/share/confluent-hub-components/plugins

當涉及到自定義寫入策略時，插件管理器對 .jars 的位置非常挑剔。 只需刪除所有這些行並適當更新插件路徑即可解決問題