[英]Add @timestamp field in ElasticSearch with Python
I'm using Python to add entries in a local ElasticSearch (localhost:9200) 我正在使用Python在本地ElasticSearch中添加条目(localhost:9200)
Currently, I use this method: 目前,我使用这种方法:
def insertintoes(data):
"""
Insert data into ElasicSearch
:param data: dict
:return:
"""
timestamp = data.get('@timestamp')
logstashIndex = 'logstash-' + timestamp.strftime("%Y.%m.%d")
es = Elasticsearch()
if not es.indices.exists(logstashIndex):
# Setting mappings for index
mapping = '''
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": false
},
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"norms": false,
"type": "text"
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"type": "keyword"
}
},
"norms": false,
"type": "text"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date",
"include_in_all": true
},
"@version": {
"type": "keyword",
"include_in_all": true
}
}
}
}
}
'''
es.indices.create(logstashIndex, ignore=400, body=mapping)
es.index(index=logstashIndex, doc_type='system', timestamp=timestamp, body=data)
data
is a dict
structure with a valid @timestamp
defined like this data['@timestamp'] = datetime.datetime.now()
data
是一个dict
结构,其有效的@timestamp
定义为此data['@timestamp'] = datetime.datetime.now()
@timestamp
data['@timestamp'] = datetime.datetime.now()
The problem is, even if there is a timestamp value in my data, Kibana doesn't show the entry in «discovery» field. 问题是,即使我的数据中存在时间戳值,Kibana也不会在“发现”字段中显示该条目。 :( :(
Here is an example of a full entry in ElasicSearch: 以下是ElasicSearch中完整条目的示例:
{ "_index": "logstash-2017.06.25", "_type": "system", "_id": "AVzf3QX3iazKBndbIkg4", "_score": 1, "_source": { "priority": 6, "uid": 0, "gid": 0, "systemd_slice": "system.slice", "cap_effective": "1fffffffff", "exe": "/usr/bin/bash", "hostname": "ns3003395", "syslog_facility": 9, "comm": "crond", "systemd_cgroup": "/system.slice/cronie.service", "systemd_unit": "cronie.service", "syslog_identifier": "CROND", "message": "(root) CMD (/usr/local/rtm/bin/rtm 14 > /dev/null 2> /dev/null)", "systemd_invocation_id": "9228b6c72e6a4624a1806e4c59af8d04", "syslog_pid": 26652, "pid": 26652, "@timestamp": "2017-06-25T17:27:01.734453" } }
As you can see, there IS a @timestamp
field but it doesn't seems to be what Kibana expects. 正如您所看到的,有一个@timestamp
字段,但它似乎不是Kibana所期望的。
And don't know what to do to make my entries visible in Kibana. 并且不知道如何使我的条目在Kibana中可见。
Any idea ? 任何的想法 ?
Elasticsearch is not recognizing @timestamp as a date, but as a string. Elasticsearch不会将@timestamp识别为日期,而是将其视为字符串。 If your data['@timestamp'] is a datetime object, you can try to convert it to a ISO string, which is automatically recognized, try: 如果您的数据['@ timestamp']是一个日期时间对象,您可以尝试将其转换为自动识别的ISO字符串,请尝试:
timestamp = data.get('@timestamp').isoformat()
timestamp should now be a string, but in ISO format timestamp现在应该是一个字符串,但是采用ISO格式
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.