Logstash output is incorrect

Question

I am new to logstash and elasticsearch. I am using logstash to read db updates and store into elasticsearch for fast searching. Following is my logstash configuration file(countries.conf).

    input {
      jdbc {
        jdbc_driver_library => "/home/vagrant/postgresql-9.4-1201.jdbc4.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "jdbc:postgresql://192.168.10.123:5432/myDB"
        jdbc_user => "myuser"
        jdbc_password => "mypassword"
        schedule => "* * * * *"
        statement_filepath => "/home/vagrant/countries.sql"
        last_run_metadata_path => "/home/vagrant/logstash/countries.log"
      }
    }
    output {
        elasticsearch {
            index => "myIndex"
            document_type => "countries"
            document_id => "%{id}"
            hosts => "localhost:9200"
        }
        stdout { codec => json_lines }
    }

And My countries.sql file is as follows

  SELECT json.id as id,
    row_to_json(json.*) AS _source
   FROM (
        SELECT id, created, modified, name, capital, iso_alpha2, iso_alpha3 
        FROM countries
  ) as json

I run config file using following command

sudo /opt/logstash/bin/logstash -f /home/vagrant/countries.conf

Output of above command on stdout is as follows:-

Settings: Default pipeline workers: 1
Pipeline main started
{"_id":6,"_source":{"type":"json","value":"{\"id\":6,\"created\":\"2013-02-07T10:11:00\",\"modified\":\"2016-04-29T11:15:40.329\",\"name\":\"Andorra\",\"capital\":\"Andorra la Vella\",\"iso_alpha2\":\"AD\",\"iso_alpha3\":\"AND\"}"},"@version":"1","@timestamp":"2016-05-02T10:08:00.931Z"}

As you can see in above output my json string in _source field is changed. Ideally it should be like below

{"_id":6,"_source":{\"id\":6,\"created\":\"2013-02-07T10:11:00\",\"modified\":\"2016-04-29T11:15:40.329\",\"name\":\"Andorra\",\"capital\":\"Andorra la Vella\",\"iso_alpha2\":\"AD\",\"iso_alpha3\":\"AND\"},"@version":"1","@timestamp":"2016-05-02T10:08:00.931Z"}

Logstash is changing my json string. It is adding type:"json" an extra field and adding my actual json sting in value field. I crossed check in db. My SQL query is correctly returning json string in format which i need.

Can someone please let me know exactly what I am missing? or Can guide me in right direction?

Thanks in Advance!

Answer 1

Check the logs in elasticsearch. I think mostly as you are using codec => json_lines in stdout, this is the reason why type=>json is getting appended. Elasticsearch records wont have the type field.

If the logs in elasticsearch also has type=>json, use mutate to remove the field.

filter {
  mutate {
    remove_field => [ "type" ]
  }
}