logstash不将日志推送到AWS Elasticsearch

Question

I am trying to push my logs from logstash to elasticsearch but its failing. here is my logstash.conf file :

input {
            file {
                    path => "D:/shweta/ELK_poc/test3.txt"
                    start_position => "beginning"
                    sincedb_path => "NUL"
                    ignore_older => 0
                }}

    output {
        elasticsearch {
            hosts => [ "https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com" ]
            index => "testindex4-5july"
            document_type => "test-file"
        }
    } 

The ES endpoint that i have provided in hosts is open , so there should not be an access isssue, but it still gives following error:

_[2018-07-05T13:59:05,753][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/, :path=>"/"}_
_[2018-07-05T13:59:05,769][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/][Manticore::ResolutionFailure] This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server (search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com)"}_

I am stuck here. But when i downloaded ES and installed it in my machine and ran it locally , replacing hosts with: hosts => [ "localhost:9200" ] ,in output , it worked all good pushing data to local es:

I tried a lot of ways but not able to resolve the issue , can anyone please help. I don't want to give localhost but AWS ES domain endpoint. Any hints or leads will be highly appreciated

Thanks in advance Shweta

Answer 1

In my opinion, you simply need to explicitly add the port 443 and it will work. I think the elasticsearch output plugin automatically uses port 9200 if no port is explicitly given.

    elasticsearch {
        hosts => [ "https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:443" ]
        index => "testindex4-5july"
        document_type => "test-file"
    }

An alternative would be to not add the port but specify ssl => true as depicted in the official AWS ES docs

    elasticsearch {
        hosts => [ "https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com" ]
        index => "testindex4-5july"
        document_type => "test-file"
        ssl => true
    }