[英]How to add a field to kibana via logstash
我正在使用 python-logstash 来写入 logstash。 它提供了添加额外字段的选项,但问题是所有字段都在“消息”字段下。
我必须承认这个解决方案对我不起作用: 如何向 logstash/kibana 添加自定义字段?
我的 python 脚本如下所示:
LOGGER = logging.getLogger('python-logstash-logger')
LOGGER.setLevel(logging.INFO)
#LOGGER.addHandler(logstash.LogstashHandler(127.0.0.1, 5000, version=1))
LOGGER.addHandler(logstash.TCPLogstashHandler('127.0.0.1', 5000, version=1))
LOGGER.error('python-logstash: test logstash error message.')
LOGGER.info('python-logstash: test logstash info message.')
LOGGER.warning('python-logstash: test logstash warning message.')
# add extra field to logstash message
extra = {
'test_string': 'python version: ' + repr(sys.version_info),
'test_boolean': True,
'test_dict': {'a': 1, 'b': 'c'},
'test_float': 1.23,
'test_integer': 123,
'test_list': [1, 2, '3'],
}
LOGGER.info("python-logstash: test extra fields", extra=extra)
我的logstath配置文件是:
input {
beats {
port => 5044
}
stdin { codec => plain }
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
我想要的只是创建我的自定义字段,例如,从额外变量中的键创建“test_string”。 正如我所说,所有额外的变量都位于“消息”字段中,而不是我希望该 dict 中的每个键都成为 kibana 中的一个字段。 如何做到这一点?
另外,我从 logstash 收到以下错误(我在我的 powershell 中看到它):
[ERROR][logstash.codecs.json ][main] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'mestamp': was expecting ('true', 'false' or 'null')
这可能是由于损坏的令牌看起来像这样:
我知道令牌 @version : 1 可能来自我的 logstashHandler,但是 TIMESTAMP 来自哪里以及如何修复该令牌?
************************////// 更新 //////////********* ************************
我认为所有字段都出现在“消息”字段中的唯一原因是那个损坏的令牌。 如何修复那个 mestamp”令牌?它来自哪里?我没有在我的 python 或 logstash 代码中设置它。
当我使用mutate
插件时,它似乎工作正常。 这是我的logstash config file
如果您还有问题,请告诉我
input {
http {
}
}
filter {
mutate {
add_field => { "test_string" => "Python version 1" }
}
}
output {
stdout {
# codec => {rubydebug}
}
elasticsearch {
hosts=> ["localhost:9200"]
index => "so-test1"
}
}
这是我在 Kibana 中看到的
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "so-test1",
"_type" : "_doc",
"_id" : "XOUei28B--Dy_XuABlDq",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"test_string" : "Python version 1", **<== test string that I appended**
"@timestamp" : "2020-01-09T16:23:17.734Z",
"host" : "0:0:0:0:0:0:0:1",
"message" : "hello", **<=== message the I sent**
"headers" : {
"request_path" : "/",
"postman_token" : "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
"http_accept" : "*/*",
"http_host" : "localhost:8080",
"request_method" : "POST",
"cache_control" : "no-cache",
"content_type" : "text/plain",
"content_length" : "5",
"http_version" : "HTTP/1.1",
"connection" : "keep-alive",
"accept_encoding" : "gzip, deflate",
"http_user_agent" : "PostmanRuntime/7.21.0"
}
}
}
]
}
}
这是我在Logstash console
上Logstash console
{
"@version" => "1",
"test_string" => "Python version 1", **<== test_string that I added in mutate filter**
"@timestamp" => 2020-01-09T16:23:17.734Z,
"host" => "0:0:0:0:0:0:0:1",
"message" => "hello", **<=== the message that I sent through POSTMAN**
"headers" => {
"request_path" => "/",
"postman_token" => "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
"http_accept" => "*/*",
"http_host" => "localhost:8080",
"request_method" => "POST",
"cache_control" => "no-cache",
"content_type" => "text/plain",
"content_length" => "5",
"http_version" => "HTTP/1.1",
"connection" => "keep-alive",
"accept_encoding" => "gzip, deflate",
"http_user_agent" => "PostmanRuntime/7.21.0"
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.