[英]How to specify document version in elasticsearch pipeline?
I currently use an ingest node pipeline which looks like this: 我目前使用一个接收节点管道,如下所示:
{
"my-pipeline": {
"description": "pipeline for my filebeat",
"processors": [
{
"json": {
"field": "message",
"add_to_root": true,
"on_failure": [
{
"rename": {
"field": "message",
"target_field": "originalMessage",
"ignore_missing": true
}
},
{
"set": {
"field": "indexName",
"value": "pipeline-errors"
}
},
{
"set": {
"field": "indexType",
"value": "pipeline-error"
}
},
{
"rename": {
"field": "@timestamp",
"target_field": "errorTimestamp",
"ignore_missing": true
}
}
]
}
},
{
"remove": {
"field": "@timestamp",
"ignore_failure": true
}
},
{
"remove": {
"field": "message",
"ignore_failure": true
}
},
{
"script": {
"inline": "ctx._index = ctx.indexName; ctx._type=ctx.indexType; if (ctx.docVersion != null) {ctx._version = ctx.docVersion; ctx._version_type='external'}"
}
},
{
"remove": {
"field": "indexName",
"ignore_failure": true
}
},
{
"remove": {
"field": "indexType",
"ignore_failure": true
}
}
]
}
}
This pipeline is used simply unbox a log forwarded by filebeat. 此管道仅用于取消由filebeat转发的日志。 In the script processor i look for the 'indexName' and 'indexType' fields and assign it to '_index' and '_type' respectively. 在脚本处理器中,我查找'indexName'和'indexType'字段,并分别将其分配给'_index'和'_type'。 Since i need to take the version into account, a 'version' field is included in the log (but this is optional as some logs does not contain the version). 由于我需要考虑版本,因此日志中包含“版本”字段(但这是可选的,因为某些日志不包含该版本)。
Using this pipeline triggers: 使用此管道触发:
org.elasticsearch.index.mapper.MapperParsingException: Cannot generate dynamic mappings of type [_version] for [_version]
at org.elasticsearch.index.mapper.DocumentParser.createBuilderFromFieldType(DocumentParser.java:656) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:805) ~[elasticsearch-5.5.0.jar:5.5.0]
What i've tried so far ( updated 09-16 ): 我到目前为止所尝试的内容( 更新时间为09-16 ):
How do i specify/use external versioning in elasticsearch documents when using ingester node pipelines? 使用ingester节点管道时,如何在elasticsearch文档中指定/使用外部版本控制? if this is not possible through pipelines' script processor, what are the options to use an external version when working with filebeat-to-elasticsearch in such a way that older version of the document gets rejected? 如果通过管道的脚本处理器无法实现这一点,那么在使用文件绑定到弹性搜索时使用外部版本的选项有哪些选择,以使文档的旧版本被拒绝?
Update 10-24-2017 Seems that this is a feature that does not exist with the current elasticsearch version (5.6 in my case). 更新10-24-2017似乎这是当前elasticsearch版本不存在的功能(在我的情况下为5.6)。 As per checking in the code , the IndexRequest in the pipeline execution service does not include any reference to the document version nor version type thus defaulting to an internal version. 根据代码中的检查,管道执行服务中的IndexRequest不包括对文档版本或版本类型的任何引用,因此默认为内部版本。 Perhaps this can be added as a feature in future elasticsearch releases. 也许这可以作为未来弹性搜索版本中的一项功能添加。
The following variables are available through the ctx map: _index, _type, _id, _version, _routing, _parent, _now and _source. 通过ctx map可以获得以下变量:_index,_type,_id,_version,_routing,_parent,_now和_source。 You can access the original source for a field as ctx._source.field-name. 您可以将字段的原始源作为ctx._source.field-name访问。
It looks the script is trying to access a document field named "version" via ctx.version but that maps to ctx._version. 看起来脚本试图通过ctx.version访问名为“version”的文档字段,但是映射到ctx._version。
The internal doc value should be retrieved as ctx._source.version , can you try that? 内部doc值应该作为ctx._source.version检索,你能试试吗?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.