How to talk to aws elasticsearch service using elastic java client?

Question

I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this endpoint(Note that there is no port specified) I can get the expected

{
  status: 200,
  name: "Mastermind",
  cluster_name: "xxxx",
  version: {
    number: "1.5.2",
    build_hash: "yyyyyy",
    build_timestamp: "2015-04-27T09:21:06Z",
    build_snapshot: false,
    lucene_version: "4.10.4"
  },
  tagline: "You Know, for Search"
}

The question is how do I get this through the elasticsearch java client without a port number? The sample code I get is

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300));

If I use this code and just replace "host1" with my endpoint, I'll get "NoNodeAvailableException"

ps: The java client version I'm using is 2.0.0.

Edit I finally decided to go with Jest , a 3rd party REST client. But what Brooks answered below is also very helpful - AWS do use port 80 for http and 443 for https. The blocker for me was the firewall I guess.

Edit2 The AWS ES service documentation explicitly says:
The service supports HTTP on port 80, but does not support TCP transport.

Answer 1

Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.

So, to demonstrate, try this...

curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'

Or

curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'

It should respond with: {"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}

Check in Kibana and you'll see it's there.

So, then in your code, it should be:

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));

Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.

Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
    }
  ]
}

NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....

Answer 2

Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,

Elastic Transport client on AWS Managed ElasticSearch 1

There is also a discussion in the AWS forum regarding the transport protocol. Here is the link

What is the port for the transport protocol ?

Answer 3

After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java

Apart from properly configuring access to you AWS ES (ie dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).

Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861