Unable to use elasticsearch sink connector (kafka-connect)

Question

I'm currently trying to start an elasticsearch sink connector on a kafka-connect cluster (distributed mode) This cluster is deployed in kubernetes using the helm charts provided by confluent with some tweaks in it. Here is relevants parts :

For values.yaml

configurationOverrides:
  "plugin.path": "/usr/share/java,/usr/share/confluent-hub-components"
  "key.converter": "org.apache.kafka.connect.storage.StringConverter"
  "value.converter": "org.apache.kafka.connect.json.JsonConverter"
  "key.converter.schemas.enable": "false"
  "value.converter.schemas.enable": "false"
  "internal.key.converter": "org.apache.kafka.connect.json.JsonConverter"
  "internal.value.converter": "org.apache.kafka.connect.json.JsonConverter"
  "config.storage.replication.factor": "3"
  "offset.storage.replication.factor": "3"
  "status.storage.replication.factor": "3"
  "security.protocol": SASL_SSL
  "sasl.mechanism": SCRAM-SHA-256

And for the kube cluster part :

releases:
  - name: kafka-connect
    tillerless: true
    tillerNamespace: qa3-search
    chart: ../charts/cp-kafka-connect
    namespace: qa3-search
    values:
      - replicaCount: 2
      - configurationOverrides:
          config.storage.topic: kafkaconnectKApp_connect-config_private_json
          offset.storage.topic: kafkaconnectKApp_connect-offsets_private_json
          status.storage.topic: kafkaconnectKApp_connect-statuses_private_json
          connect.producer.client_id: "connect-worker-producerID"
          groupId: "kafka-connect-group-ID"
          log4j.root.loglevel: "INFO"
          bootstrap_servers: "SASL_SSL://SOME_ACCESSIBLE_URL:9094"
          client.security.protocol: SASL_SSL
          client.sasl.mechanism: SCRAM-SHA-256
      - prometheus:
          jmx:
            enabled: false
      - ingress:
          enabled: true
          hosts:
            - host: kafka-connect.qa3.k8s.XXX.lan
              paths:
                - /
      - cp-schema-registry:
          url: "https://SOME_ACCESSIBLE_URL"

Then I am loading the elasticsearch sink connector as such :

curl -X POST -H 'Content-Type: application/json' http://kafka-connect.qa3.k8s.XXX.lan/connectors -d '{
"name": "similarads3",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
"topics": "SOME_TOPIC_THAT_EXIST",
"topic.index.map": "SOME_TOPIC_THAT_EXIST:test_similar3",
"connection.url": "http://vqa38:9200",
"batch.size": 1,
"type.name": "similads",
"key.ignore": true,
"errors.log.enable": true,
"errors.log.include.messages": true,
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "SOME_ACCESSIBLE_URL",
"schema.ignore": true
}
}' -vvv

More over I'm loading user and password for brokers auth via environment variable, and I'm pretty sure it is connected with rights ACL...

What is troubling me, is that there is no index creation when the connector starts, and there is no error what so ever in kafka-connect's logs... And it says everything has started

Starting connectors and tasks using config offset 68

When running a curl on /connectors/similarads3/status, everything is running, without errors.

So it seems like I overlooked something, but I can't figure out what is missing. When I check consumers lag on this particular topics, it seems like no messages where consumed ever.

If there is not enough information, I'm able to provide more. Does someone have an idea ?

EDIT : I should have mentioned that I tried to configure it with a topic that does not exist : again no error in logs. (I don't know how to interpret this)

EDIT 2 : This issue is solved Actually we found the issue and it appears that i did overlooked something: in order to read from a topic protected by ACLs rights, you have to provide the SASL configuration for both the connector and the sink consumer. So just duplicating the configuration prefixed with consumer. fixed this problem. However I'm still surprised that no logs can point to this.

Answer 1

We had issues trying to use the topic.index.map property. Even if you got it working there is a note in the docs that it is deprecated.

topic.index.map
This option is now deprecated. A future version may remove it completely. Please use single message transforms, such as RegexRouter, to map topic names to index names.

I'd try using the RegexRouter to accomplish this instead.

"transforms": "renameTopicToIndex",
"transforms.renameTopicToIndex.type": "org.apache.kafka.connect.transforms.RegexRouter"
"transforms.renameTopicToIndex.regex": ".*"
"transforms.renameTopicToIndex.replacement": "test_similar3"