AWS Kinesis Firehose - using Index Rotation (Elasticsearch)

Question

I have set up a new AWS Kinesis Firehose stream and I'd like to create a new index on a weekly basis.

For that, I should use Index Rotation setting when configuring the stream.

But, do I have to the new index every weekend for the next upcoming week? If not (hopefully not), how does Firehose knows what mapping to use? does it use the mapping defined in the index that I specified in the Index setting?

More over, lets say I have old data, can I make Firehose to create an index with the relevant timestamp according to the dates in specified in my old data?

Thank you !

Answer 1

Have you tried to consider creating index template in elastic search. Thay way new indexes will pick the mapping defined in index template.

Refer to following link for details

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

Answer 2

Well, apparently the answer is yes, but in a bad way.

If Firehose pushes data to a new index, that is not pre-defined with a mapping, that data is ingested to Elasticsearch and a mapping is created automatically for you.

This is really bad.

You should auto create an index 1-2 hours prior the event. I'll post a Lambda function and its configuration for doing that automatically.