Kafka 連接器到 map 主題鍵作為 ElasticSearch 中的文檔 ID

Question

我正在嘗試 map Kafka 主題鍵作為文檔 ID，同時使用 Kafka 接收器連接器索引到彈性搜索。 但我得到一個例外，關鍵是 null

如以下示例中所述，我希望將“AKA-25”作為文檔 ID，但我正在進行的轉換失敗。 如果我從值中獲取任何其他字段，就像我嘗試使用“empId”一樣，它似乎工作正常。 但我的要求是 map 是關鍵。

有人可以建議嗎？

這是連接器配置：

{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "doc",
  "tasks.max": "1",
  "key.ignore": "false",
  "schema.ignore": "true",
  "key.converter.schemas.enable": "false",
  "value.converter.schemas.enable": "false",
  "connection.url": "http://elasticsearch:9200",
  "topics": "emp_change",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.json.JsonConverter",
  "transforms": "addSuffix,InsertKey,extractKey,dropPrefix",
  "transforms.extractKey.field": "key",
  "transforms.extractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
  "transforms.InsertKey.fields": "key",
  "transforms.InsertKey.type": "org.apache.kafka.connect.transforms.ValueToKey"
  "transforms.dropPrefix.regex": "emp_change(.*)",
  "transforms.dropPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter", 
  "transforms.dropPrefix.replacement": "$1", 
  "transforms.addSuffix.replacement": "employee",
  "transforms.addSuffix.regex": ".*",
  "transforms.addSuffix.type": "org.apache.kafka.connect.transforms.RegexRouter",
  "transforms.createKey.fields": "key"
}

Kafka 主題消息如下所示：

{
    "topic": "EmpChange",
    "key": "AKA-25",
    "value": {
      "eventType": "EMP_CHG",
      "empId": 1001,
      "empName": "AKASH"
    },
    "partition": 0,
    "offset": 0
  }

這是我得到的例外：

org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:560)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: Key is used as document id and can not be null.
    at io.confluent.connect.elasticsearch.DataConverter.convertKey(DataConverter.java:79)
    at io.confluent.connect.elasticsearch.DataConverter.convertRecord(DataConverter.java:160)
    at io.confluent.connect.elasticsearch.ElasticsearchWriter.tryWriteRecord(ElasticsearchWriter.java:285)
    at io.confluent.connect.elasticsearch.ElasticsearchWriter.write(ElasticsearchWriter.java:270)
    at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put(ElasticsearchSinkTask.java:169)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
    ... 10 more

Answer 1

我認為你的配置有錯誤。 如您的數據所示，您的消息有一個密鑰（ "key": "AKA-25", ）：

{
    "topic": "EmpChange",
    "key": "AKA-25",
    "value": {
      "eventType": "EMP_CHG",
      "empId": 1001,
      "empName": "AKASH"
    },
    "partition": 0,
    "offset": 0
  }

但是，您有一個單消息轉換，它指示 Kafka Connect 將消息值中的字段key復制到消息的鍵中

  "transforms.InsertKey.fields": "key",
  "transforms.InsertKey.type": "org.apache.kafka.connect.transforms.ValueToKey"

但是由於key不是消息value中的字段，因此轉換將在 Kafka 消息鍵中插入 NULL，因此您得到的錯誤

Caused by: org.apache.kafka.connect.errors.ConnectException: Key is used as document id and can not be null.

要使用的正確配置應該是這樣的：

{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "doc",
  "tasks.max": "1",
  "key.ignore": "false",
  "schema.ignore": "true",
  "value.converter.schemas.enable": "false",
  "connection.url": "http://elasticsearch:9200",
  "topics": "PPAP_C01_ApexUserChange",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "transforms": "addSuffix,dropPrefix",
  "transforms.dropPrefix.regex": "PPAP_C01_ApexUserChange(.*)",
  "transforms.dropPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter", 
  "transforms.dropPrefix.replacement": "$1", 
  "transforms.addSuffix.replacement": "users1",
  "transforms.addSuffix.regex": ".*",
  "transforms.addSuffix.type": "org.apache.kafka.connect.transforms.RegexRouter"
}

你所擁有的和這個更新的配置之間的差異是：

• 移除ValueToKey和ExtractField$Key轉換
• 刪除孤立createKey轉換配置
• 將密鑰轉換器更改為org.apache.kafka.connect.storage.StringConverter並刪除對應schemas.enable