简体   繁体   中英

kafka-connect-elasticsearch How to route multiple topics to same elasticsearch index in same connector?

Trying to create elasticsearch sink connector with following config, the creation is successful but when a message is produced on "my.topic.one", ES sink connector fails while trying to create an index with name "my.topic.one": "Could not create index 'my.topic.one'" (User that I am using to connect to ES does not have create index permission intentionally). Why is it trying to create a new index and how to get the connector to index to previously created "elasticsearch_index_name"?

{
            "type.name": "_doc", 
            "tasks.max": "1", 
            "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", 
            "connection.url": "http://elasticsearch-service:9200", 
            "behavior.on.null.values": "delete", 
            "key.ignore": "false", 
            "write.method": "upsert", 
            "key.converter": "org.apache.kafka.connect.storage.StringConverter", 
            "key.converter.schemas.enable": "false", 
            "value.converter": "org.apache.kafka.connect.storage.StringConverter", 
            "value.converter.schemas.enable": "false", 
            "topics": "my.topic.one,my.topic.two", 
            "transforms": "renameTopic",
            "transforms.renameTopic.type": "org.apache.kafka.connect.transforms.RegexRouter", 
            "transforms.renameTopic.regex": ".*",
            "transforms.renameTopic.replacement": "elasticsearch_index_name"
    } 

UPDATE: ES sink connector throws error even if I use just one topic in "topics" attribute and same topic name in "renameTopic.regex" like below, rest all attributes same.

"topics": "my.topic.one",
"transforms.renameTopic.regex": "my.topic.one"

Adding following property to ES sink connector config, solved the issue at hand:

"auto.create.indices.at.start": "false"

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM