How to send json events to ES with Logstash Plugin for Amazon ES?

Question

I am new to ES. Trying to send json events to ES with https://github.com/awslabs/logstash-output-amazon_es However, when I give below configuration it does not recognize any events?

   input {
        file {
            path => "C:/Program Files/logstash-2.3.1/transactions.log"
            start_position => beginning
            codec => "json_lines"
        }
    }
    filter {
       json {
            source => "message"
       }
    }
    
    output {
        amazon_es {
            hosts => ["endpoint"]
            region => "us-east-1"
            codec => json
            index => "production-logs-%{+YYYY.MM.dd}"
            
        }
    }

I am running it in debug mode but there is nothing in the log Also do I need to create the index before I start sending the events from Logstash?

The below config works somehow, however it does not recognize any json fields

input {
    file {
        path => "C:/Program Files/logstash-2.3.1/transactions.log"
        start_position => beginning
    }
}

output {
    amazon_es {
        hosts => ["Endpoint"]
        region => "us-east-1"
        index => "production-logs-%{+YYYY.MM.dd}"
    }
}

Answer 1

There may be several things at play here, including:

• Logstash thinks your file has already been processed. start_position is only for files that haven't been seen before. If you're testing, set sincedb_path to /dev/null, or manually manage your registry files.
• You're having mapping problems. Elasticsearch will drop documents when the field mapping isn't correct (trying to insert a string into a numeric field, etc). This should be shown in the elasticsearch logs, if you can get to them on AWS.
• debug is very verbose. If you're really getting nothing, then you're not receiving any input. See the first bullet item.
• adding a stdout{} output is a good idea until you get things working. This will show you what logstash is sending to elasticsearch.