Unable to send data to AWS elastic search instance using logstash

Question

I am trying to send data to AWS elastic search end point using logstash that is installed on my local machine.

The logstash conf file looks like this

input {
     file {
        path => "/path/log.txt"
      }
}
output {
   amazon_es {
        hosts => ["https://search-abclostashtrial-5jdfc43oqql7qsrhfgbvwewku.us-east-2.es.amazonaws.com"]
    action => "index"   
    region => "us-east-2"   
    index => "trial"
    ssl => true

    }
}

The Elastic search Access policy looks like this

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-2:0415721453395:domain/abclostashtrial/*"
    }
  ]
}

I am using logstash-output-amazon_es plugin to send the query like

sudo bin/logstash -f /path/logstash/abc.conf

And I get the following error log.

[ERROR] 2019-04-30 20:05:52.900 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil} 
[INFO ] 2019-04-30 20:05:53.165 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} 
[INFO ] 2019-04-30 20:05:58.037 [LogStash::Runner] runner - Logstash shut down.

What am I missing here ?

Answer 1

One option to start with is to create an AccessKey that has rights to write to Elasticsearch, and configure that in the output. Example:

amazon_es {
    hosts => ["vpc-xxxxxxxxx-es-yyyyyy4pywmwigwi47em.us-east-1.es.amazonaws.com"]
    region => "us-east-1"
    aws_access_key_id => 'AKIxxxxxxxxxxx'
    aws_secret_access_key => '11xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    index => "production-logindex-%{+YYYY.MM.dd}"
    }