使用 Elasticsearch Sink Connector for Kafka 从两个由下划线分隔的值字段创建文档 ID

Question

I am trying to load records from a Kafka topic to Elasticsearch using the Elasticsearch Sink Connector, but I'm struggling to construct the document ids the way I would like them. I would like the document id that is written to Elasticsearch to be a composition of two values separated by underscore from my kafka topic's message.

For example:

My Kafka topic value has the following Avro schema:

{
  "type": "record",
  "name": "SampleValue",
  "namespace": "com.abc.test",
  "fields": [
    {
      "name": "value1",
      "type": [
        "null",
        {
          "type": "int",
          "java-class": "java.lang.Integer"
        }
      ],
      "default": null
    },
    {
      "name": "value2",
      "type": [
        "null",
        {
          "type": "int",
          "java-class": "java.lang.Integer"
        }
      ],
      "default": null
    },
    {
      "name": "otherValue",
      "type": [
        "null",
        {
          "type": "int",
          "java-class": "java.lang.Integer"
        }
      ],
      "default": null
    }
  ]
}

I would like the document id that is written to Elasticsearch to be the combined values of value1 and value2 separated by an underscore. If the given value in avro looked like

{"value1": {"int": 123}, "value2": {"int": 456}, "value3": {"int": 0}}

then I would like the document id for Elasticsearch to be 123_456 .

I can't figure out the correct way to chain transformations in my connector config to create a key that is composed of two values separated by an underscore.

Answer 1

I don't think there is a Single Message Transform out of the box that will do what you want.

You can either write your own, using the Transform API, or you can use a stream processor such as Kafka Streams or ksqlDB.