Using Flask to POST to Elasticsearch Hosted Service with api_key

Question

I am working on connecting a Flask API to my hosted Elasticsearch instance and I keep getting this error: elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, 'security_exception', 'missing authentication credentials for REST request

I have created an API key on the elastic account settings and have tried linking to the id and Elasticsearch endpoint, but I still get this error. Right now I am just running this on my laptop, but I am wondering what I am doing wrong with the credentials or are there more security parameters I need to configure on the website interface?

from flask import Flask, request, redirect
from datetime import datetime
from flask_cors import CORS, cross_origin
import json
import requests
from elasticsearch import Elasticsearch

es = Elasticsearch(cloud_id='xxxxxxxx', api_key=('test2','XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=='))

@app.route('/queryUpdate/', methods=['POST'])
@cross_origin()
def update_query():
    def processItineraries(j):
        j = json.loads(j.decode('utf-8'))
        j['destination'] = j['to']
        j['origin'] = j['from']
        j.pop('from', None)
        j.pop('to', None)
        return json.dumps(j).encode('utf-8')

    print('update_query()')
    if request.method == "POST":
        query_obj = request.data

        #process data
        query_obj = processItineraries(query_obj)

        print(query_obj)

        # send data to kenesis
        response = es.index(index="searches", body=query_obj)
        print('*-*-' * 15)
        print(response['result'])
        print('*-*-' * 15)
        return 'success'

Answer 1

I was able to resolve this issue because the api key that I created was for the Elastic Cloud platform and not specifically for the instance itself. I needed to create an api key by going into Kibana > Dev Tools and then executing the below query.

POST /_security/api_key
{
  "name": "my-api-key2",
  "expiration": "1d", 
  "role_descriptors": { 
    "role-a": {
      "cluster": ["all"],
      "index": [
        {
          "names": ["searches*"],
          "privileges": ["write"]
        }
      ]
    }
  }
}

From here an api key will be created and can then be saved as a python object or environmental variable as seen below and then the Elasticsearch() is fed the api id and api key as a tuple.

api = {
  "id" : "xxxxxxxxxxxxxxxxxx",
  "name" : "my-api-key2",
  "expiration" : 1602861320480,
  "api_key" : "yyyyyyyyyyyyyyyyyyy"
}

es = Elasticsearch(['https://xxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.aws.found.io:9243'], api_key=(api['id'],api['api_key']))

Elasticsearch Create Api Key documentation