用mongo-connector连接亚马逊的Elasticsearch

Question

I'm trying to use mongo-connector with Amazon's new hosted Elasticsearch .

I believe that it's not using normal Elasticsearch Shield etc, but instead it is using Amazon's rather complex authentication scheme.

(related GH issue: https://github.com/mongodb-labs/mongo-connector/issues/337 )

Answer 1

I guess you've tried that, but the easiest way is to modify your access policy to "Allow Open Access to the domain" in your Amazon Elasticsearch configuration. Of course, that means your ES cluster is open to the world, so that might be OK for testing purposes, but not in production.

You can also choose "Allow access from specific IPs" , which will "secure" your cluster in a way that only requests from the specified IP addresses will be accepted. For my personal use, I've found this mode to be the easiest to work with, because you get a secured access (provided you're the only one on your IP address) and you don't need to go through the whole AWS signing request stuff. Although, if you're in VPC, that option won't work obviously.

With any of the above access policies configured on your search domain, you don't need to do anything special on the mongo-connector side, you can simply start your mongo-connector with the standard command-line and the replication will work out of the box.

mongo-connector -m localhost:27017 -t search-<yourdomain>.us-west-2.es.amazonaws.com:80 -d elastic_doc_manager

Now, if your search domain is secured with Allow/Deny access to specific AWS account or IAM users" , you will need to sign your requests , which implies that elastic_doc_manager.py will need to somehow be modified to do so. The easiest option would be to write a dedicated doc manager by extending elastic_doc_manager.py into elastic_aws_doc_manager.py and use the Amazon-provided Python boilerplate code to sign each request made by the ES doc manager, that's trivial but still a lot of work.

However, since mongo_connector es_doc_manager.py uses the official Elasticsearch Python client, and someone had the nice idea to tweak it and make it work for signing requests to AWS Elasticsearch , writing a custom elastic_aws_doc_manager.py doc manager becomes much more trivial.

I firmly believe this should even be built into the main elastic_doc_manager.py and offer a configuration option when working with the AWS Elasticsearch service.