[英]Trying to index kafka topic in Elasticsearch with Kafka Connect
I want to index a topic from kafka in avro to elasticsearch format but I have problems with my timestamp field to be recognized by elasticsearch as date format field. 我想将afro中的主题从kafka索引为elasticsearch格式,但是我的时间戳字段有问题,elasticsearch将其识别为日期格式字段。
I have used the following configuration for the connector. 我对连接器使用了以下配置。
{
"name": "es-sink-barchart-10",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"value.converter.schema.registry.url": "http://localhost:8081",
"connection.url": "http://localhost:9200",
"type.name":"type.name=kafka-connect",
"topics": "exchange_avro_01",
"topic.index.map": "exchange_avro_01:exchange_barchart",
"key.ignore": "true"
}
}
The original field is bigint type and I want the target field to be date type with any valid format with elasticsearch. 原始字段是bigint类型,我希望目标字段是具有Elasticsearch的任何有效格式的日期类型。 I have defined a dynamic template to try to solve it in the following way: 我定义了一个动态模板,以尝试通过以下方式解决它:
curl -XPUT "http://localhost:9200/_template/kafkaconnect/" -H 'Content-Type: application/json' -d'
{
"index_patterns": "exchange*",
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"kafka-connect": {
"dynamic_templates": [
{
"dates": {
"match_mapping_type": "long",
"match": "TIME",
"mapping": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
}
}
}
]
,
"properties": {
"CLOSE": {
"type": "double"
},
.
.
.
}
}
}
}
}'
When I load the connector described above nothing is indexed to elasticsearch. 当我加载上述连接器时,没有任何东西索引到elasticsearch。
Any help? 有什么帮助吗?
If your source is a bigint then presumably it's an epoch. 如果您的消息来源是bigint,那么大概是一个时代。 If it's an epoch, then this won't work: 如果是一个纪元,那么这将不起作用:
"mapping": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
}
because you're telling Elasticsearch that the date format is yyyy-MM-dd HH:mm:ss
(which it isn't). 因为您要告诉Elasticsearch日期格式为yyyy-MM-dd HH:mm:ss
(不是)。
So instead, try this (omitting your custom mapping for the moment; get this working first and then add that back in): 因此,请尝试以下操作(暂时省略您的自定义映射;首先使它工作,然后再添加回去):
{
"index_patterns": "exchange*",
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"kafka-connect": {
"dynamic_templates": [
{
"dates": {
"match": "TIME",
"mapping": {
"type": "date"
} } } ] } } }
Also ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection 另请参阅: https : //www.elastic.co/guide/zh-CN/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection
nothing is indexed to elasticsearch. 没有东西被索引到elasticsearch。
Check the Kafka Connect worker log and the Elasticsearch log for any errors. 检查Kafka Connect工作程序日志和Elasticsearch日志是否有任何错误。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.