[英]How to format a timestamp from DynamoDB for Elasticsearch/Kibana?
I'm pushing DynamoDB rows into a Elasticsearch cluster.我正在将DynamoDB行推送到Elasticsearch集群中。 The date fields are unix timestamps and not recognized by Kibana as a date.日期字段是unix 时间戳, Kibana无法将其识别为日期。
I read about Elasticsearch mapping types and found this post , but don't know where to implement the mapping in my Lambda script:我阅读了Elasticsearch 映射类型并找到了这篇文章,但不知道在哪里在我的Lambda脚本中实现映射:
/* ... requires and config ... */
exports.handler = (event, context, callback) => {
event.Records.forEach((record) => {
var dbRecord = JSON.stringify(record.dynamodb);
postToES(dbRecord, context, callback);
});
};
function postToES(doc, context, lambdaCallback) {
var req = new AWS.HttpRequest(endpoint);
req.method = 'POST';
req.path = path.join('/', esDomain.index, esDomain.doctype);
req.region = esDomain.region;
req.headers['presigned-expires'] = false;
req.headers['Host'] = endpoint.host;
req.body = doc;
// Maybe here?
var signer = new AWS.Signers.V4(req , 'es');
signer.addAuthorization(creds, new Date());
var send = new AWS.NodeHttpClient();
send.handleRequest(req, null, function(httpResp) {
var respBody = '';
httpResp.on('data', function (chunk) {
respBody += chunk;
});
httpResp.on('end', function (chunk) {
lambdaCallback(null,'Lambda added document ' + doc);
});
}, function(err) {
console.log('Error: ' + err);
lambdaCallback('Lambda failed with error ' + err);
});
}
Elasticsearch document弹性搜索文档
{
_index: "posts",
_type: "post",
_id: "6YKF2AAV06RSSRrzv6R-",
_version: 1,
found: true,
_source: {
ApproximateCreationDateTime: 1499922960,
Keys: {
id: {
S: "7asda8b0-628a-11e7-9e5e-25xyc7179dx7"
}
},
NewImage: {
posted_at: {
N: "1499922995401"
},
id: {
S: "7asda8b0-628a-11e7-9e5e-25xyc7179dx7"
}
},
SequenceNumber: "2442423900000000003279639454",
SizeBytes: 221,
StreamViewType: "NEW_AND_OLD_IMAGES"
}
}
Dynamoose Schema Dynamoose 架构
var Schema = dynamoose.Schema;
var s = new Schema({
id: {
type: String,
hashKey: true,
required: true
},
posted_at: {
type: Date,
required: true
}
});
module.exports = dynamoose.model('posts', s);
Example: in my DynamoDB table I've the field posted_at
.示例:在我的 DynamoDB 表中,我有posted_at
字段。 The content is a unix timestamp.内容是一个unix时间戳。 In Kiabana it's indexed as在 Kiabana 中,它被索引为
NewImage.posted_at.N
(type: string, searchable, analyzed) and NewImage.posted_at.N
(类型:字符串、可搜索、分析)和NewImage.posted_at.N.keyword
(type: string, searchable, aggregateable) NewImage.posted_at.N.keyword
(类型:字符串、可搜索、可聚合) I'm confused by the N
and type: string
.我对N
和type: string
感到困惑。
Any ideas?有任何想法吗? Thanks!谢谢!
Ok it turns out that the N
is there to denote the DynamoDB attribute type (ie N
stands for Number
).好的,结果证明N
表示DynamoDB 属性类型(即N
代表Number
)。
The problem is that the number gets stringified and thus indexed as a string in ES (ie what you currently see in your mapping).问题是数字被字符串化,因此在 ES 中被索引为字符串(即您当前在映射中看到的内容)。
We can get around this using a dynamic template definition.我们可以使用动态模板定义来解决这个问题。 First delete your index in ES and the corresponding index pattern in Kibana.首先删除你在 ES 中的索引和 Kibana 中对应的索引模式。 Then run this command:然后运行这个命令:
curl -XPUT localhost:9200/_template/post_template -d '{
"template": "posts",
"mappings": {
"post": {
"dynamic_templates": [
{
"dates": {
"path_match": "NewImage.posted_at.N",
"mapping": {
"type": "date"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
]
}
}
}'
Finally you can reindex your data through Dynamoose and you should be able to find a date field in Kibana afterwards.最后,您可以通过 Dynamoose 重新索引您的数据,之后您应该能够在 Kibana 中找到日期字段。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.