将数据从 AWS Elasticsearch 移动到 S3

Question

I have an application pumping logs to an AWS OpenSearch (earlier Elasticsearch) cluster. I want to move old logs to S3 to save cost and still be able to read the logs (occasionally).

One approach I can think of is writing a cron job that reads the old indexes, writes them (in text format) to the s3 and deletes the indexes. This also requires keeping day wise indexes. Is there a more efficient/better way?

Answer 1

You can use the manual snapshots approach to backup your indexes to s3: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html

Another option as suggested toward the end of the first link is to use a tool named Curator within lambda that will handle the index rotation: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/curator.html

Depending on your use case UltraWarm could be the best approach, in case you want those logs to be searchable later on without the need of manual restores, that will be required in case you go with the first two options I have listed: https://aws.amazon.com/blogs/aws/general-availability-of-ultrawarm-for-amazon-elasticsearch-service/

Answer 2

There is one tool elasticdump

# Export ES data to S3 (using s3urls)
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input=http://production.es.com:9200/my_index \
  --output "s3://${bucket_name}/${file_name}.json"