I am using python-logstash in order to write to logstash. It offers the option to add extra fields but problem is that all fields are under the 'message' field.
I must admit that this solution doesn't work for me: How do I add a custom field to logstash/kibana?
My python script looks like this:
LOGGER = logging.getLogger('python-logstash-logger')
LOGGER.setLevel(logging.INFO)
#LOGGER.addHandler(logstash.LogstashHandler(127.0.0.1, 5000, version=1))
LOGGER.addHandler(logstash.TCPLogstashHandler('127.0.0.1', 5000, version=1))
LOGGER.error('python-logstash: test logstash error message.')
LOGGER.info('python-logstash: test logstash info message.')
LOGGER.warning('python-logstash: test logstash warning message.')
# add extra field to logstash message
extra = {
'test_string': 'python version: ' + repr(sys.version_info),
'test_boolean': True,
'test_dict': {'a': 1, 'b': 'c'},
'test_float': 1.23,
'test_integer': 123,
'test_list': [1, 2, '3'],
}
LOGGER.info("python-logstash: test extra fields", extra=extra)
And my logstath confing file is:
input {
beats {
port => 5044
}
stdin { codec => plain }
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
All I want is to create my custom fields, eg, 'test_string' from the keys in the extra variable. As I said, all that extra variable lands in the 'message' field not while I want each key in that dict to become a field in kibana. How to accomplish this?
Plus, I'm getting a following error from the logstash(I see it in my powershell):
[ERROR][logstash.codecs.json ][main] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'mestamp': was expecting ('true', 'false' or 'null')
This is probably due to the broken token that looks like this:
I know that the token @version : 1 comes probably from my logstashHandler, but where that TIMESTAMP comes from and how to fix that token?
************************////// update //////////******************************
I think the only reason why all the fields land in the 'message' field is that broken token. How to fix that mestamp" token? And where does it come from? I do not set it in my python or logstash code.
It seems works fine when I use mutate
plugin. Here is my logstash config file
Let me know if you still have questions
input {
http {
}
}
filter {
mutate {
add_field => { "test_string" => "Python version 1" }
}
}
output {
stdout {
# codec => {rubydebug}
}
elasticsearch {
hosts=> ["localhost:9200"]
index => "so-test1"
}
}
and here is what I see in my Kibana
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "so-test1",
"_type" : "_doc",
"_id" : "XOUei28B--Dy_XuABlDq",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"test_string" : "Python version 1", **<== test string that I appended**
"@timestamp" : "2020-01-09T16:23:17.734Z",
"host" : "0:0:0:0:0:0:0:1",
"message" : "hello", **<=== message the I sent**
"headers" : {
"request_path" : "/",
"postman_token" : "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
"http_accept" : "*/*",
"http_host" : "localhost:8080",
"request_method" : "POST",
"cache_control" : "no-cache",
"content_type" : "text/plain",
"content_length" : "5",
"http_version" : "HTTP/1.1",
"connection" : "keep-alive",
"accept_encoding" : "gzip, deflate",
"http_user_agent" : "PostmanRuntime/7.21.0"
}
}
}
]
}
}
and here is what I see on Logstash console
{
"@version" => "1",
"test_string" => "Python version 1", **<== test_string that I added in mutate filter**
"@timestamp" => 2020-01-09T16:23:17.734Z,
"host" => "0:0:0:0:0:0:0:1",
"message" => "hello", **<=== the message that I sent through POSTMAN**
"headers" => {
"request_path" => "/",
"postman_token" => "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
"http_accept" => "*/*",
"http_host" => "localhost:8080",
"request_method" => "POST",
"cache_control" => "no-cache",
"content_type" => "text/plain",
"content_length" => "5",
"http_version" => "HTTP/1.1",
"connection" => "keep-alive",
"accept_encoding" => "gzip, deflate",
"http_user_agent" => "PostmanRuntime/7.21.0"
}
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.