I'm using confluentinc/cp-kafka-connect docker image. I'm trying to send JSON file to kafka, with elasticsearch id.
{"_id":10000725, "_source": {"createdByIdentity":"tu_adminn","createdBy":"Admin Testuser"}}
here is my connector
{
"name": "test-connector",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "andrii",
"key.ignore": "false",
"schema.ignore": "true",
"connection.url": "http://elasticsearch:9200",
"type.name": "test-type",
"name": "elasticsearch-sink"
}
}
When i'm using key.ignore = true it's generates some weird id. How can i pass exactly my id and source?
If you specify key.ignore=true
then Kafka Connect will use a composite key of your message's kafka topic, partition, and offset -- this is the "weird id" that you're seeing.
If you want to use your own ID for the created Elasticsearch document, you can set key.ignore=false
and Kafka Connect will use the key of the Kafka message as the ID.
If your Kafka message does not have the appropriate key for what you want to do, you will need to set it. One option is to use something like KSQL :
CREATE STREAM target AS SELECT * FROM source PARTITION BY _id
Disclaimer: I work for Confluent, the company behind the open-source KSQL project
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.