Logstash filter parse json file result a double fields

Question

I am using the latest ELK (Elasticsearch 1.5.2 , Logstash 1.5.0, Kibana 4.0.2) I have a question that

sample .json

{ "field1": "This is value1", "field2": "This is value2" }

longstash.conf

input {
   stdin{ }
}

filter {
        json {
                source => "message"
                add_field =>
                {
                        "field1" => "%{field1}"
                        "field2" => "%{field2}"
                }
        }
}

output {
   stdout { codec => rubydebug }
   elasticsearch {
      host => "localhost"
      index => "scan"
   }
}

Output:

{
       "message" => "{ \"field1\": \"This is value1\", \"field2\": \"This is value2\" }",
      "@version" => "1",
    "@timestamp" => "2015-05-07T06:02:56.088Z",
          "host" => "myhost",
        "field1" => [
        [0] "This is value1",
        [1] "This is value1"
    ],
        "field2" => [
        [0] "This is value2",
        [1] "This is value2"
    ]
}

My question is 1) why the field result appear double in the result? 2) If there is nested array , how is it should reference in the logstash configure?

Thanks a lot!

..Petera

Answer 1

I think you have misunderstood what the json filter does. When you process a field through the json filter it will look for field names and corresponding values.

In your example, you have done that with this part:

filter {
        json {
                source => "message"

Then you have added a field called "field1" with the content of field "field1", since the field already exists you have just added the same information to the field that was already there, it has now become an array:

                add_field =>
                {
                        "field1" => "%{field1}"
                        "field2" => "%{field2}"
                }
        }
}

If you simplify your code to the following you should be fine:

filter {
        json {
                source => "message"
        }
}

I suspect your question about arrays becomes moot at this point, as you probably don't need the nested array, and therefore, won't need to address it, but in case you do, I believe you can do this like so:

[field1][0]
[field1][1]