Logstash标签层次结构

Question

I'm installing Logstash, ElasticSearch & Kibana, and am trying to implement a hierarchy of logs that can be rendered as a tree.

eg Take a hierarchy such as: web->site1->access , web->site1->error , mail->list1->bounces etc

Then have a web interface that can list this as a tree:

web 
  | - site1
        | - access
        | - error
mail
  | -- list1
         | -- bounces

So users can quickly find the access log for site1.

I can write the web part (or add a custom panel to Kibana), but am not sure how to configure Logstash/Elasticsearch to store this.

Just with ElasticSearch, I tried adding fields like:

POST /test/logs
{
    "types": ["web", "site1", "access"],
    "message":"access log line 1"
}
POST /test/logs
{
    "types": ["mail", "list1", "bounces"],
    "message":"access log line 1"
}
...

But the types array gets split up (tokenized?), so when I do a facets query I get back a list of all the individual array values combined. Which is probably what you'd want if these were tags, but not in my case.

Is storing a hierarchy of tags possible in Logstash / ElasticSearch, and if so how?

Answer 1

That's an odd way of doing things.

You could add new fields, eg:

hostname: mail
site: site1
maillist: list1
log: access

Then be more specific in your query:

hostname:web AND site:site1 AND log:access

Provide a few prepared dashboards that users can tweak values to get what they need.

Otherwise you need to use nested field syntax and probably conditionals

Eg (untested)

filter{
  mutate{
    add_field => [ ["%{servertype}"]["%{hostname}"]["%{site}"]["%{log}"], "some value" ]
  }
}

If you are sending in your logs in JSON format from the start, then you can create nested fields just by adding more nested JSON objects. eg:

{
"message": "GET /interestingpage?param1=value1 HTTP/1.1 404",
"tags": [
    "tag1",
    "tag2",
    "tag3",
    "valve"
],
"@timestamp": "2013-10-24T20:23:46.222Z",
"field3": "value3",
"field2": "value2",
"level": "ERROR",
"http": {
    "request_protocol": "HTTP/1.1",
    "response_headers": {},
    "request_querystring": "param1=value1",
    "remote_user": null,
    "request_headers": {
        "Referer": null,
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36"
    },
    "response_size": 985,
    "response_status": 404,
    "request_parameters": {
        "param1": [
            "value1"
        ]
    },
    "request_uri": "/interestingpage",
    "remote_host": "192.168.1.208",
    "server_name": "192.168.1.67",
    "request_method": "GET",
    "response_duration": 27,
    "cookies": {}
},
"hostname": "precise64",
"field1": "value1",
"@version": "1",
"user": "tomcat7"
}

Nested fields in kibana are . (dot) separated in the field list. But you can use the field filter to narrow down to find what you need.