如何从 Kinesis firehose 将 stream 数据传输到自托管 elasticsearch 集群？

Question

I deployed a ES cluster to AWS EKS and it can be access through application load balancer. I have created an index in the cluster. I am going to deploy a Kinesis firehose to use http_endpoint as the target to stream data to ES cluster.

But how can I configure what http verb firehose uses to put data to ES endpoint? For example, my ES endpoint is http://xxxx.com/myindex and I'd like the firehose to send POST request to this endpoint. How can I configure it to use POST rather than PUT ?

Answer 1

Kinesis Firehose doesn't give you that much control over what the HTTP requests look like, unfortunately, since there are only a handful of configuration options for the HTTP Endpoint destination.

Kinesis sends HTTP requests according to their own specification, and expects responses to look in a specific way. You can find the specification here.

This specification is not compatible with Elasticsearch's Index API as you discovered. On a side note, you will probably want to use the Bulk API for performance reasons, since Kinesis will buffer events and deliver multiple ones in one request.

So, you will most likely need something in-between Kinesis and your Elasticsearch cluster to handle the differences in regards to the HTTP requests and responses. This could be a simply nginx proxy, for instance, or it could be something like an API Gateway which invokes a Lambda function. This Lambda function then sends a request to your index' Bulk API and constructs a response for Kinesis according to the HTTP specification.

I hope that helps.