无法通过Python访问ElasticSearch AWS

Question

I'm trying to access ElasticSearch AWS from my localhost through Python (I can access it through my browser).

from elasticsearch import Elasticsearch
ELASTIC_SEARCH_ENDPOINT = 'https://xxx'
es = Elasticsearch([ELASTIC_SEARCH_ENDPOINT])

I'm receiving this error:

ImproperlyConfigured('Root certificates are missing for certificate validation. Either pass them in using the ca_certs parameter or install certifi to use it automatically.',)

How can I access it? I have not configured any certificate, I only liberated the IPs that can access ElasticSearch Service.

Answer 1

for python 3.5 install certifi and use ca_certs=certifi.where() this will pass the certificates

import certifi
from elasticsearch import Elasticsearch

host = 'https://###########.ap-south-1.es.amazonaws.com'

es = Elasticsearch([host], use_ssl=True, ca_certs=certifi.where())

Answer 2

elasticsearch-py doesn't ship with default set of root certificates. To have working SSL certificate validation you need to either specify your own as ca_certs or install certifi which will be picked up automatically.

from elasticsearch import Elasticsearch

# you can use RFC-1738 to specify the url
es = Elasticsearch(['https://user:secret@localhost:443'])

# ... or specify common parameters as kwargs

# use certifi for CA certificates
import certifi

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True 
)

# SSL client authentication using client_cert and client_key

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True,
    ca_certs='/path/to/cacert.pem',
    client_cert='/path/to/client_cert.pem',
    client_key='/path/to/client_key.pem',
)

https://elasticsearch-py.readthedocs.io/en/master/

Answer 3

I did it this way and it worked:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

host = 'YOURHOST.us-east-1.es.amazonaws.com'
awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')

es = Elasticsearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)
print(es.info())

Answer 4

You can also use boto3 to generate tempory access key & secret key.

import boto3

region = 'ap-southeast-2'
service = 'es'
session = boto3.Session()
credentials = session.get_credentials()

awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,region, service,session_token=credentials.token)

es = Elasticsearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html