Logstash无法在JSON中正确转换

Question

Following is my json log file

[
    {
        "error_message": " Failed to get line from input file (end of file?).", 
        "type": "ERROR", 
        "line_no": "2625", 
        "file": "GTFplainText.c", 
        "time": "17:40:02", 
        "date": "01/07/16", 
        "error_code": "GTF-00014"
    }, 
    {
        "error_message": " Bad GTF plain text file header or footer line. ", 
        "type": "ERROR", 
        "line_no": "2669", 
        "file": "GTFplainText.c", 
        "time": "17:40:02", 
        "date": "01/07/16", 
        "error_code": "GTF-00004"
    }, 
    {
        "error_message": " '???' ", 
        "type": "ERROR", 
        "line_no": "2670", 
        "file": "GTFplainText.c", 
        "time": "17:40:02", 
        "date": "01/07/16", 
        "error_code": "GTF-00005"
    }, 
    {
        "error_message": " Failed to find 'event source'/'product detail' records for event source '3025188506' host event type 1 valid", 
        "type": "ERROR", 
        "line_no": "0671", 
        "file": "RGUIDE.cc", 
        "time": "15:43:48", 
        "date": "06/07/16", 
        "error_code": "RGUIDE-00033"
    }
]

According to my understanding As the log is already in json, We do not need filter section in logstash configuration. Following is my logstash config

input {
  file{
    path => "/home/ishan/sf_shared/log_json.json"
    start_position => "beginning"
    codec => "json"
  }
}

and the output configuration is

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}

But it seems like the data is not going into ES, as I am not able to see the data when I query the index. What am I missing?

Answer 1

I think the problem is that the json codec expects a full json message on one line and won't work with a message on multiple lines.

A possible work around would be to use the multiline codex and use the json filter .
The configuration for the multiline codec would be:

multiline {
  pattern => "]"
  negate => "true"
  what => "next"
}

All the lines that do not begin with ] will be regrouped with the next line, so you'll have one full json document to give to the json filter.