如何通过logstash向kibana添加字段

Question

I am using python-logstash in order to write to logstash. It offers the option to add extra fields but problem is that all fields are under the 'message' field.

I must admit that this solution doesn't work for me: How do I add a custom field to logstash/kibana?

My python script looks like this:

LOGGER = logging.getLogger('python-logstash-logger')
LOGGER.setLevel(logging.INFO)
#LOGGER.addHandler(logstash.LogstashHandler(127.0.0.1, 5000, version=1))
LOGGER.addHandler(logstash.TCPLogstashHandler('127.0.0.1', 5000, version=1))
LOGGER.error('python-logstash: test logstash error message.')
LOGGER.info('python-logstash: test logstash info message.')
LOGGER.warning('python-logstash: test logstash warning message.')

# add extra field to logstash message
extra = {
    'test_string': 'python version: ' + repr(sys.version_info),
    'test_boolean': True,
    'test_dict': {'a': 1, 'b': 'c'},
    'test_float': 1.23,
    'test_integer': 123,
    'test_list': [1, 2, '3'],
}

LOGGER.info("python-logstash: test extra fields", extra=extra)

And my logstath confing file is:

input {
  beats {
    port => 5044
  }
  stdin { codec => plain }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

All I want is to create my custom fields, eg, 'test_string' from the keys in the extra variable. As I said, all that extra variable lands in the 'message' field not while I want each key in that dict to become a field in kibana. How to accomplish this?

Plus, I'm getting a following error from the logstash(I see it in my powershell):

[ERROR][logstash.codecs.json     ][main] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'mestamp': was expecting ('true', 'false' or 'null')

This is probably due to the broken token that looks like this:

I know that the token @version : 1 comes probably from my logstashHandler, but where that TIMESTAMP comes from and how to fix that token?

************************////// update //////////******************************

I think the only reason why all the fields land in the 'message' field is that broken token. How to fix that mestamp" token? And where does it come from? I do not set it in my python or logstash code.

Answer 1

It seems works fine when I use mutate plugin. Here is my logstash config file Let me know if you still have questions

 input {

    http {                                                                                                        

    }   

 }

  filter {
     mutate {
        add_field => { "test_string" => "Python version 1" }
     }
   }

output {
    stdout {
  #     codec => {rubydebug}
    }   
    elasticsearch {

      hosts=> ["localhost:9200"]
      index => "so-test1"
    }   
}

and here is what I see in my Kibana

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "so-test1",
        "_type" : "_doc",
        "_id" : "XOUei28B--Dy_XuABlDq",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "test_string" : "Python version 1",  **<== test string that I appended**
          "@timestamp" : "2020-01-09T16:23:17.734Z",
          "host" : "0:0:0:0:0:0:0:1",
          "message" : "hello",     **<=== message the I sent**
          "headers" : {
            "request_path" : "/",
            "postman_token" : "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
            "http_accept" : "*/*",
            "http_host" : "localhost:8080",
            "request_method" : "POST",
            "cache_control" : "no-cache",
            "content_type" : "text/plain",
            "content_length" : "5",
            "http_version" : "HTTP/1.1",
            "connection" : "keep-alive",
            "accept_encoding" : "gzip, deflate",
            "http_user_agent" : "PostmanRuntime/7.21.0"
          }
        }
      }
    ]
  }
}

and here is what I see on Logstash console

{
       "@version" => "1",
    "test_string" => "Python version 1",  **<== test_string that I added in mutate filter**
     "@timestamp" => 2020-01-09T16:23:17.734Z,
           "host" => "0:0:0:0:0:0:0:1",
        "message" => "hello",    **<=== the message that I sent through POSTMAN**
        "headers" => {
           "request_path" => "/",
          "postman_token" => "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
            "http_accept" => "*/*",
              "http_host" => "localhost:8080",
         "request_method" => "POST",
          "cache_control" => "no-cache",
           "content_type" => "text/plain",
         "content_length" => "5",
           "http_version" => "HTTP/1.1",
             "connection" => "keep-alive",
        "accept_encoding" => "gzip, deflate",
        "http_user_agent" => "PostmanRuntime/7.21.0"
    }
}