简体   繁体   English

kafka-connect-elasticsearch 如何将多个主题路由到同一连接器中的同一 elasticsearch 索引?

[英]kafka-connect-elasticsearch How to route multiple topics to same elasticsearch index in same connector?

Trying to create elasticsearch sink connector with following config, the creation is successful but when a message is produced on "my.topic.one", ES sink connector fails while trying to create an index with name "my.topic.one": "Could not create index 'my.topic.one'" (User that I am using to connect to ES does not have create index permission intentionally).尝试使用以下配置创建 elasticsearch 接收器连接器,创建成功,但是当在“my.topic.one”上生成消息时,ES 接收器连接器在尝试创建名称为“my.topic.one”的索引时失败:“无法创建索引'my.topic.one'”(我用来连接到 ES 的用户没有故意创建索引权限)。 Why is it trying to create a new index and how to get the connector to index to previously created "elasticsearch_index_name"?为什么要尝试创建新索引以及如何让连接器索引到以前创建的“elasticsearch_index_name”?

{
            "type.name": "_doc", 
            "tasks.max": "1", 
            "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", 
            "connection.url": "http://elasticsearch-service:9200", 
            "behavior.on.null.values": "delete", 
            "key.ignore": "false", 
            "write.method": "upsert", 
            "key.converter": "org.apache.kafka.connect.storage.StringConverter", 
            "key.converter.schemas.enable": "false", 
            "value.converter": "org.apache.kafka.connect.storage.StringConverter", 
            "value.converter.schemas.enable": "false", 
            "topics": "my.topic.one,my.topic.two", 
            "transforms": "renameTopic",
            "transforms.renameTopic.type": "org.apache.kafka.connect.transforms.RegexRouter", 
            "transforms.renameTopic.regex": ".*",
            "transforms.renameTopic.replacement": "elasticsearch_index_name"
    } 

UPDATE: ES sink connector throws error even if I use just one topic in "topics" attribute and same topic name in "renameTopic.regex" like below, rest all attributes same.更新:即使我在“主题”属性中仅使用一个主题并且在“renameTopic.regex”中使用相同的主题名称,ES sink 连接器也会引发错误,如下所示,rest 所有属性都相同。

"topics": "my.topic.one",
"transforms.renameTopic.regex": "my.topic.one"

Adding following property to ES sink connector config, solved the issue at hand:将以下属性添加到 ES sink 连接器配置,解决了手头的问题:

"auto.create.indices.at.start": "false"

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 kafka-connect-elasticsearch:如何将 elasticsearch 与消费者组同步? - kafka-connect-elasticsearch: How to sync elasticsearch with consumer group? kafka-connect-elasticsearch:将消息存储为预定义索引的格式 - kafka-connect-elasticsearch: storing messages as format of predefined index kafka-connect-elasticsearch:如何发送文件删除? - kafka-connect-elasticsearch: how to send deletes of documents? kafka-connect-elasticsearch:当使用“write.method”作为 upsert 时,是否可以在 kafka 主题上使用相同的 AVRO object 来发送部分文档? - kafka-connect-elasticsearch: When using “write.method” as upsert, is it possible to use same AVRO object on kafka topic to send partial document? kafka-connect-elasticsearch:如何根据 Kafka 主题的 header 中的某个值删除文档 - kafka-connect-elasticsearch: How to delete document based on certain value in header of the Kafka topic 无法将主题映射到 kafka elasticsearch 连接器中的指定索引 - unable to map topics to specified index in kafka elasticsearch connector 卡夫卡连接器 Elasticsearch topic.regex - Kafka connector Elasticsearch topics.regex Elasticsearch接收器仅使用kafka-connect-elasticsearch +时间戳SMT仅获得新消息,而不接收前一条消息 - Elasticsearch sink only get new messages and not the previous one using kafka-connect-elasticsearch + timestamp SMT Kafka Connect Elasticsearch Connector的消息顺序 - Message order with Kafka Connect Elasticsearch Connector 用于Kafka Connect的Elasticsearch连接器-偏移量和时间戳 - Elasticsearch connector for Kafka Connect - offset and timestamp
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM