使用Logstash处理多个JSON事件

Question

I have the following configuration: logs from the application instances are forwarded to elastic search using filebeat and logstash.

   Apps
+--------+
| +--------+
| | +--------+     +----------+    +----------+
| | |   +--- |     |          |    |          |
+ | |   |file| --> | logstash | -->| elastic  |
  + |   |beat|     |   (1)    |    | search   |
    +--------+     +----------+    +----------+
                         |               |
             (not avail) X               | (query)
                         V               |
                   +----------+          V
                   |          |      +------+
                   | logstash |<-----| Json |
                   |   (2)    |      | file |
                   +----------+      +------+

I want to test the log processing in logstash-2, but I cannot presently implement the forwarding from logstash-1. So I tried the following: query elasticsearch and retrieve the documents's _source fields and I got some json documents like this:

{
 "@timestamp": <timestamp>,
 "@version": "1",
 "requestMethod": "PUT",
 "requestUri": "/api/endopoint",
 "servername": "myserver" 
 ....  many other fields
}
{
 "@timestamp": <timestamp>,
 "@version": "1",

}
... many other json objects

My question is, how can I process these json documents from the elasticsearch query using logstash?

I have tried to process them using the multiline codec and then the json filter, but cannot manage to make it work: Here is an attempt:

input {
  file {
    path => "events.json"
    sincedb_path => "/dev/null"
    start_position => beginning
    codec => multiline {
       pattern => "^\}"    #end of each json object
       what => "previous"
    }
  }
}

filter {
  json {
    source => "event"
  }
 }

 output {
  stdout{}
}

Answer 1

After some additional research, I have realized the multiline codec configuration was wrong. I have fixed and now I have the whole event on the message field.

input {
  file {
    path => "events.json"
    sincedb_path => "/dev/null"
    start_position => beginning
    codec => multiline {
        pattern => "^\}"    #end of each json object
        negate => true
        what => "next"
    }
  }
}

filter {
  json {
    source => "message"
  }

  mutate {
    remove_field => ["message"]
  } 
} 

output {
 stdout{}
}