如何使用logstash将json文件中的所有内容发送到elasticsearch

Question

I have test results that are being stored in json files. I then have logstash locate the file and attempt to send all of the lines to elasticsearch. Only about half of the lines are being sent and can't figure out why certain lines are being left out. For example, there will be 34 lines, but only 14 are sent.

input {
    file {
        path => "/data/*.json"
        start_position => "beginning"
    }
}

# ----------------------------------------------------------------------

filter {

    # Parse fields out of JSON message, then remove the raw JSON. 
    json {
        source => "message"
    }

}

# ----------------------------------------------------------------------

output {
    elasticsearch {
        hosts => ["host:9200", "localhost:9200"]
        index => "ct-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }

I'm not sure if there is something within the json itself that causes logstash to just skip over it, or if there's something wrong with my logstash.conf file that I posted above.

Answer 1

Logstash computes files from different types to send it to elasticsearch in Json format. In your case, a Filebeat agent with an elasticsearch output would be enough to send a json file to ES and to index it.

It would look like this using Filebeat 6.x :

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  # Paths to the logs
  paths:
    - "/your/path/to/your/logs/file.json"
  # tags to identify the logs source, .gz files excluded from the prospector
  tags: ["beats","yourtag"]
  exclude_files: ['\.gz$']

#================================ Outputs =====================================
#----------------------------- Elasticsearch output --------------------------------
output.elasticsearch:
  # The ES host & index name
  hosts: ["yourEShost:9200"]
  index: "ct-%{+YYYY.MM.dd}"