简体繁体 English

AWS Kinesis Firehose-使用索引旋转（Elasticsearch）

[英]AWS Kinesis Firehose - using Index Rotation (Elasticsearch)

原文 2016-07-31 17:21:48 6 2 amazon-web-services/ aws-sdk/ amazon-kinesis/ amazon-kinesis-firehose

I have set up a new AWS Kinesis Firehose stream and I'd like to create a new index on a weekly basis. 我已经建立了一个新的AWS Kinesis Firehose流，并且希望每周创建一个新索引。

For that, I should use Index Rotation setting when configuring the stream. 为此，我应在配置流时使用“ Index Rotation设置。

But, do I have to the new index every weekend for the next upcoming week? 但是，在接下来的下一周，我每个周末都必须使用新索引吗？ If not (hopefully not), how does Firehose knows what mapping to use? 如果不是（希望不是）， Firehose如何知道要使用什么映射？ does it use the mapping defined in the index that I specified in the Index setting? 它是否使用我在“ Index设置中指定的索引中定义的映射？

More over, lets say I have old data, can I make Firehose to create an index with the relevant timestamp according to the dates in specified in my old data? 此外，可以说我有旧数据，我可以让Firehose根据我的旧数据中指定的日期创建带有相关时间戳的索引吗？

Thank you ! 谢谢！

2 个解决方案

Have you tried to consider creating index template in elastic search. 您是否尝试过考虑在弹性搜索中创建索引模板。 Thay way new indexes will pick the mapping defined in index template. 通过这种方式，新索引将选择索引模板中定义的映射。

Refer to following link for details 详情请参阅以下链接

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

Well, apparently the answer is yes, but in a bad way. 好吧，显然答案是肯定的， 但是方式很糟糕。

If Firehose pushes data to a new index, that is not pre-defined with a mapping, that data is ingested to Elasticsearch and a mapping is created automatically for you. 如果Firehose将数据推送到未使用映射预先定义的新索引，则该数据将被摄取到Elasticsearch并自动为您创建映射。

This is really bad. 这真的很糟糕。

You should auto create an index 1-2 hours prior the event. 您应该在活动开始前1-2小时自动创建索引。 I'll post a Lambda function and its configuration for doing that automatically. 我将发布一个Lambda函数及其配置，以自动执行此操作。