Import JSON Files into Logstash + Elasticsearch + Kibana

Question

So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc...

Now, I've been toying with the Logstash + Elasticsearch + Kibana3 stack, and I'd love to find a way to see those logs in Kibana. My question is, is there a way to make Logstash import these kind of files, or would I have to write a custom input plugin for it? I've searched around and for what I've seen, plugins are written in Ruby, a language I don't have experience with.

Answer 1

Logstash is a very good tool for processing dynamic files.

Here is the way to import your json file into elasticsearch using logstash:

configuration file:

input 
{
    file 
    {
        path => ["/path/to/json/file"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        exclude => "*.gz"
    }
}

filter 
{
    mutate
    {
        replace => [ "message", "%{message}" ]
        gsub => [ 'message','\n','']
    }
    if [message] =~ /^{.*}$/
    {
        json { source => message }
    }

}

output
{ 
  elasticsearch {
    protocol => "http"
    codec => json
    host => "localhost"
    index => "json"
    embedded => true
  }

    stdout { codec => rubydebug }
}

example of json file:

{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}

Note the json need to be in one line. if you want to parse a multiline json file, replace relevant fields in your configuration file:

   input 
{   
    file 
    {
        codec => multiline
        {
            pattern => '^\{'
            negate => true
            what => previous                
        }
        path => ["/opt/mount/ELK/json/*.json"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        exclude => "*.gz"
    }
}

filter 
{
    mutate
    {
        replace => [ "message", "%{message}}" ]
        gsub => [ 'message','\n','']
    }
    if [message] =~ /^{.*}$/ 
    {
        json { source => message }
    }

}

Answer 2

Logstash is just a tool for converting various kinds of syslog files into JSON and loading them into elasticsearch (or graphite, or... ).

Since your files are already in JSON, you don't need logstash. You can upload them directly into elasticsearch using curl.

See Import/Index a JSON file into Elasticsearch

However, in order to work well with Kibana, your JSON files need to be at a minimum.

1. Flat - Kibana does not grok nested JSON structs. You need a simple hash of key/value pairs.

2. Have a identifiable timestamp.

What I would suggest is looking the JSON files logstash outputs and seeing if you can massage your JSON files to match that structure. You can do this in any language you like that supports JSON. The program jq is very handy for filtering json from one format to another.

Logstash format - https://gist.github.com/jordansissel/2996677

jq - http://stedolan.github.io/jq/

Answer 3

Logstash can import different formats and sources as it provides a lot of plugins. There are also other log collector and forwarder tools that can send logs to logstash such as nxlog , rsyslog, syslog-ng, flume, kafka, fluentd, etc. From what I've heard most people use nxlog on windows (though it works on linux equally well) in combination with the ELK stack because of its low resource footprint. (Disclaimer: I'm affiliated with the project)