using elasticsearch filter in logstash pipeline

Question

I'm using the elasticsearch filter in my logstash pipeline. I correctly find the result using :

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
    }
  }
}

as you can see, Im using "query_template" which is :

{
    "query": {
      "query_string": {
       "query": "class:DPAPI AND request.aw:%{[aw]}"
      }
    },
   "_source": ["end_point", "vittorio"]
 }

that tells elastichsearch to look up the log with that specific class that match "aw" with the DPAPIINTERNAL log.

Perfect! but now that i found the result, i want to add some field from it and attach them to my DPAPIINTERNAL log, for instance, i want to take "end_point" and add it in the new key "vittorio" inside my log.

This is not happening and I don't understand why.

here is the log that i'm looking at using the query:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "dp_audit-2017.02.16",
        "_type": "logs",
        "_id": "AVpHoPHPuEPlW12Qu",
        "_score": 1,
        "_source": {
          "svc": "dp-1.1",
          "request": {
            "method": "POST|PATCH|DELETE",
            "aw": "prova",
            "end_point": "/bank/6311",
            "app_instance": "7D1-D233-87E1-913"
          },
          "path": "/home/vittorio/Documents/dpapi1.json",
          "@timestamp": "2017-02-16T15:53:33.214Z",
          "@version": "1",
          "host": "Vito",
          "event": "bank.add",
          "class": "DPAPI",
          "ts": "2017-01-16T19:20:30.125+01:00"
        }
      }
    ]
  }
}

Answer 1

Your need to specify the fields parameter in your elasticsearch filter, like this:

elasticsearch {
  hosts => "10.1.10.16"
  index => "dp_audit-2017.02.16"
  query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
  fields => { "[request][end_point]" => "vittorio" }
}

Note that since end_point is a nested field, you need to modify the _source in your query template like this:

"_source": ["request.end_point"]

Answer 2

the problem is simply that you don't have to specify the "new" field using the query_template.

"_source": ["request"] # here you specify the field you want from the query result.

and then

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
      fields => {"request" => "new_key"} # here you add the fields and will tell elastich filter to put request inside new_key
    }
  }
}

That worked for me!