Making signed requests to AWS Elasticsearch Service with typescript

Question

What I need is to load streaming data from DynamoDB into Amazon Elasticsearch Service using a typescript lambda.

The streaming data arrives as expected but I haven't been able to figure out how to authenticate an HTTP request with a signature. AWS appears to have some private APIs for signing and issuing HTTP requests which are referenced in this JS code (for loading data to ES from S3):

/*
 * Add the given document to the ES domain.
 * If all records are successfully added, indicate success to lambda
 * (using the "context" parameter).
 */
function postDocumentToES(doc, context) {
    var req = new AWS.HttpRequest(endpoint);

    req.method = 'POST';
    req.path = path.join('/', esDomain.index, esDomain.doctype);
    req.region = esDomain.region;
    req.body = doc;
    req.headers['presigned-expires'] = false;
    req.headers['Host'] = endpoint.host;

    // Sign the request (Sigv4)
    var signer = new AWS.Signers.V4(req, 'es');
    signer.addAuthorization(creds, new Date());

    // Post document to ES
    var send = new AWS.NodeHttpClient();
    send.handleRequest(req, null, function(httpResp) {
        var body = '';
        httpResp.on('data', function (chunk) {
            body += chunk;
        });
        httpResp.on('end', function (chunk) {
            numDocsAdded ++;
            if (numDocsAdded === totLogLines) {
                // Mark lambda success.  If not done so, it will be retried.
                console.log('All ' + numDocsAdded + ' log records added to ES.');
                context.succeed();
            }
        });
    }, function(err) {
        console.log('Error: ' + err);
        console.log(numDocsAdded + 'of ' + totLogLines + ' log records added to ES.');
        context.fail();
    });
}

source: https://github.com/aws-samples/amazon-elasticsearch-lambda-samples/blob/master/src/s3_lambda_es.js

But when writing in TS I can't seem to import the Signers.V4 or NodeHttpClient libs, I guess because they're "private APIs".

Another confusing factor is that in this documentation we are told that the aws-sdk automatically authenticates outgoing requests and that we shouldn't need to do it ourselves: https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Could anyone please shed some insight on how to use TypeScript to authenticate/sign and send http requests with AWS Lambda credentials to an AWS Service, like Elasticsearch?

Answer 1

Here is my implementation for Elasticsearch client compatible with Amazon Elasticsearch AWS4 signed requests:

import { Client } from "@elastic/elasticsearch";
import * as AWS4 from "aws4";
import { Connection } from "@elastic/elasticsearch";
import { ConnectionOptions } from "@elastic/elasticsearch/lib/Connection";
import * as http from "http";
import { Readable } from "stream";

interface RequestOptions extends http.ClientRequestArgs {
    asStream?: boolean;
    body?: string | Buffer | Readable | null;
    querystring?: string;
}

class AwsEsConnection extends Connection {
    constructor(opts?: ConnectionOptions) {
        super(opts);
    }

    getBodyString(body?: string | Buffer | Readable | null): string | null {
        if (!body) {
            return body as null;
        }

        if (typeof body === "string" || body instanceof String) {
            return body as string;
        }

        if (body instanceof Buffer) {
            return body.toString();
        }

        if (body instanceof Readable) {
            throw new Error("Haven't implemented stream handling!!");
        }

        return body;
    }

    public request(
        params: RequestOptions,
        callback: (
            err: Error | null,
            response: http.IncomingMessage | null
        ) => void
    ): http.ClientRequest {
        const body = this.getBodyString(params.body);
        const opts = {
            method: params.method,
            host: params.host,
            path: `${params.path}?${params.querystring}`,
            service: "es",
            region: "eu-west-1",
            body: body,
            headers: params.headers
        };

        AWS4.sign(opts);

        params.headers = opts.headers;
        params.body = opts.body;

        return super.request(params, callback);
    }
}

export class ElasticClientFactory {
    constructor(
        private esClusterBaseUrl: string,
    ) {}

    create(): Client {
        return new Client({
            node: this.esClusterBaseUrl,
            Connection: AwsEsConnection
        });
    }
}

Answer 2

I've also had good results with this package, which does the same thing under the hood, essentially:

https://www.npmjs.com/package/aws-elasticsearch-connector