Kafka connect弹性搜索ID创建多个字段不起作用

Question

I am asking this question as there was no answer in the original case: Elastic Kafka Connector, ID Creation .

I have a similar situation.

Elastic search table to create a record for a single field, but not for multiple fields when request sent through kafkaconnect.

Getting exception "Key is used as document id and can not be null" in elastic search.

My Connector Configurations:

{
 "name": "test-connector33",
 "config": {
 "connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
 "tasks.max": "1",
 "topics": "test-connector33",
 "connection.url": "http://localhost:9200",
 "type.name": "aggregator",
 "schema.ignore": "true",
 "topic.schema.ignore": "true",
  "topic.key.ignore": "false",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": "false", 
 "key.converter": "org.apache.kafka.connect.json.JsonConverter",
 "key.converter.schemas.enable": "false",
 "key.ignore":"false",
 "name": "test-connector33",
"transforms": "InsertKey,extractKey",
"transforms.InsertKey.type":"org.apache.kafka.connect.transforms.ValueToKey",
"transforms.InsertKey.fields":"customerId,city",
"transforms.extractKey.type":"org.apache.kafka.connect.transforms.ExtractField$Key",
"transforms.extractKey.field":"customerId,city"
}}

Any idea how to resolve this?

Thanks in advance!

Answer 1

org.apache.kafka.connect.transforms.ExtractField$Key only supports single fields.

Pretend your JSON object was a HashMap<String, Object> . You cannot find the field customerId,city , and so that map.get(field) operation returns null , therefore setting the field to be null.

If you want to send keys via the console producer, you are welcome to do that by adding --property print.key=true as a flag, then typing the key, press tab, then putting the value. If you want to echo data into the process, then you can also set --property key.separator='|' , for a vertical bar as well as add --property parse.key=true