Logstash - won't parse JSON

Question

I want to parse data to Elasticsearch using Logstash. So far this worked great but when I try to parse JSON files, Logstash just won't do ...anything. I can start Logstash without any exception but it won't parse anything.

Is there something wrong with my config? The path to the JSON file is correct.

my JSON:

{
  "stats": [
    {
      "liveStatistic": {
        "@scope": "21",
        "@scopeType": "foo",
        "@name": "minTime",
        "@interval": "60",
        "lastChange": "2011-01-11T15:19:53.259+02:00",
        "start": "2011-01-18T14:19:48.333+02:00",
        "unit": "s",
        "value": 10
      }
    },
    {
      "liveStatistic": {
        "@scope": "26",
        "@scopeType": "bar",
        "@name": "newCount",
        "@interval": "60",
        "lastChange": "2014-01-11T15:19:59.894+02:00",
        "start": "2014-01-12T14:19:48.333+02:00",
        "unit": 1,
        "value": 5
      }
    },
    ...
  ]
}

my Logstash agent config:

input {
  file {
    path => "/home/me/logstash-1.4.2/values/stats.json"
    codec => "json"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    host => localhost
    protocol =>"http"
  }
  stdout { 
    codec => rubydebug 
  }
}

Answer 1

You should add the following line to your input:

start_position => "beginning"

Also put the complete document on one line and maybe add {} around your document to make it a valid json document.

Answer 2

Okay, two things:

First, the file input is by default set to start reading at the end of the file. If you want the file to start reading at the beginning, you will need to set start_position. Example:

file {
    path => "/mypath/myfile"
    codec => "json"
    start_position => "beginning"
}

Second, keep in mind that logstash keeps a sincedb file which records how many lines of a file you have already read (so as to not parse information repeatedly!). This is usually a desirable feature, but for testing over a static file (which is what it looks like you're trying to do) then you want to work around this. There are two ways I know of.

One way is you can just make a new copy of the file every time you want to run logstash, and remember to tell logstash to read from that file. The other way is you can go and delete the sincedb file, wherever it is located. You can tell logstash where to write the sincedeb file with the sincedb_path feature.

I hope this all helped!