堆棧內存溢出
  簡體   English   中英

AWS中的Elasticsearch快照和還原

[英]Elasticsearch Snapshot & Restore In AWS

 原文  2017-07-04 13:42:06  6   2    amazon-web-services/ elasticsearch

問題描述

我正在使用AWS ES（一項托管服務）。 AWS確實每天都會執行自動備份。 我想執行類似但更頻繁的操作。

為此，我創建了一個S3存儲桶，將其注冊為ES群集中​​的存儲庫，並編寫了一個調度程序以在指定時間拍攝群集快照。

ES快照本質上是增量快照，即所有現有快照均加載到內存中，以確定要保存在當前快照中的更改。

隨着時間的推移，快照數量會增加。

我想保留特定數量的快照並刪除其他快照。 為此，我們可以編寫另一個調度程序。

但是，在運行快照創建調度程序之前，其余的將不足以還原整個群集。

有解決這個問題的好方法嗎？

請提出建議。

2 個解決方案

解決方案1
 2  2019-09-06 15:21:22

Q1.:--如何設置Amazon Elasticsearch Service手冊索引快照。

https://github.com/miztiik/AWS-Demos/tree/master/How-To/setup-manual-elasticsearch-snapshots 

S3-Bucket-Name =    xxxxxxx-es-snapshot-repo
ES-IAM-Role =       xxxxxxx-es-snapshot-role
ES-REPO-NAME=       xxxxxxx-es-snapshot-repository
ES-IAM-USER =       xxxxxxx-es-snapshot-user
ES-IAM-Policy =     xxxxxxx-es-snapshot-access
ES-POLICY=          xxxxxxx-es-allow-role
ES-DOMAIN-NAME =    xxxxxxx-waf-logs
ES-END-POINT =      https://search-xxxxxxx-waf-logs efsphsb67nsvddjxxxxxxxxx.us-east-1.es.amazonaws.com

================================================== ========================

==>快照是群集數據和狀態的備份。 狀態包括群集設置，節點信息，索引設置和分片分配。 Elasticsearch快照是增量快照，這意味着它們僅存儲自上次成功快照以來已更改的數據。 這種增量性質意味着，頻繁快照和不頻繁快照之間的磁盤使用差異通常很小。

==>快照提供了一種方便的方法來跨Amazon Elasticsearch Service域遷移數據並從故障中恢復。 自動快照是給定域中的只讀快照。 您不能使用自動快照遷移到新域。 對於遷移，您必須使用手動快照。

================================================== ========================

先決條件

================================================== ========================

  1. ElasticSearch域ES-DOMAIN-NAME = xxxxxxx-waf-logs

  2. 創建S3存儲桶-xxxxxxx-es-snapshot-repo

  3. 獲取存儲桶ARN-arn：aws：s3 ::: xxxxxxx-es-snapshot-repo

  4. IAM角色：xxxxxxx-es-snapshot-role-

注意：附加以下權限，請確保更改存儲桶ARN

================================================== ======================== 

 {
   "Version": "2012-10-17",
   "Statement": [{
       "Action": [
         "s3:ListBucket"
       ],
       "Effect": "Allow",
       "Resource": [
         "arn:aws:s3:::xxxxxxx-es-snapshot-repo"
       ]
     },
     {
       "Action": [
         "s3:GetObject",
         "s3:PutObject",
         "s3:DeleteObject"
       ],
       "Effect": "Allow",
       "Resource": [
         "arn:aws:s3:::xxxxxxx-es-snapshot-repo/*"
       ]
     }
   ]
 }

================================================== ========================

將以下信任關系附加到角色

================================================== ======================== 

 {
   "Version": "2012-10-17",
   "Statement": [{
     "Sid": "",
     "Effect": "Allow",
     "Principal": {
       "Service": "es.amazonaws.com"
     },
     "Action": "sts:AssumeRole"
   }]
 }

================================================== ========================

  1. 使用AWS CLI創建IAM用戶-xxxxxxx-es-snapshot-user在此處獲取幫助

================================================== ========================

將以下策略附加到用戶

================================================== ======================== 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::467657035428:role/xxxxxxx-es-snapshot-role"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:us-east-1:467657035428:domain/xxxxxxx-waf-logs/*"
        }
    ]
}

================================================== ========================

6.運行Linux且可以連接到ES群集的EC2-只需將其托管在與ES相同的VPC /子網中

使用IAM用戶xxxxxxx-es-snapshot-user配置的AWS CLI

================================================== ========================

注冊手動快照存儲庫

================================================== ========================

==>必須先在Amazon Elasticsearch Service中注冊快照存儲庫，然后才能進行手動索引快照。 如果您的ES域駐留在VPC內，則您的計算機必須連接到VPC才能成功注冊快照存儲庫

================================================== ========================

准備EC2客戶端以注冊我們的S3存儲庫

================================================== ========================

注意：在以下代碼中更改主機，區域和ROLE ARN以適合您的環境。

================================================== ========================

安裝一些必備軟件包

================================================== ======================== 

yum -y install python-pip

pip install requests-aws4auth

================================================== ========================

創建python文件來注冊倉庫

================================================== ======================== 

cat >/tmp/register-repo.py <<"EOF"
import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://search-xxxxxxx-waf-logs-efsphsb67nsvddjxxxxxxxxx.us-east-1.es.amazonaws.com/'
region = 'us-east-1' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository
path = '_snapshot/xxxxxxx-es-snapshot-repository' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "xxxxxxx-es-snapshot-repo",
    "region": "us-east-1",
    "role_arn": "arn:aws:iam::467657035428:role/xxxxxxx-es-snapshot-role"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)
EOF

================================================== ========================

執行文件注冊回購

================================================== ========================

chmod 700 /tmp/register-repo.py

================================================== ======================== 

python /tmp/register-repo.py

200
{"acknowledged":true}

================================================== ========================

拍攝手動快照

================================================== ========================

創建快照時，需要指定兩條信息：

快照存儲庫的名稱-例如：xxxxxxx-es-snapshot-repository快照的名稱-例如：2019-02-01

================================================== ========================

注意：快照不是即時的； 他們需要一些時間才能完成。

================================================== ======================== 

curl -XPUT 'search-xxxxxxx-waf-logs-efsphsb67nsvddjxxxxxxxxx.us-east-1.es.amazonaws.com/_snapshot/xxxxxxx-es-snapshot-repository/2019-02-28'

================================================== ========================

使用以下命令來驗證您的域的快照狀態：

================================================== ======================== 

curl -XGET 'https://search-xxxxxxx-waf-logs-efsphsb67nsvddjxxxxxxxxx.us-east-1.es.amazonaws.com/_snapshot/xxxxxxx-es-snapshot-repository/_all?pretty'

輸出：- 

{
  "snapshots" : [ {
    "snapshot" : "snapshot-name",
    "uuid" : "FciYMhzFR1iLs0I0Nb1YeA",
    "version_id" : 6040299,
    "version" : "6.4.2",
    "indices" : [ "logs", "awswaf-2019-02-13", "logstash-2019.02.06", "filebeat-6.6.0-2019.02.19", "logstash-2019.02.13", "awswaf-2019-02-21", "awswaf-2019-02-24", "logstash-2019.02.15", "logs-2019-02-15", "filebeat-6.6.0-2019.02.26", "logstash-2019.02.21", "logs-2019-02-13", "awswaf-2019-02-01", "logstash-2019.02.20", "awswaf-2019-02-07", "awswaf-2019-02-26"],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2019-02-26T13:31:59.721Z",
    "start_time_in_millis" : 1551187919721,
    "end_time" : "2019-02-26T16:24:48.806Z",
    "end_time_in_millis" : 1551198288806,
    "duration_in_millis" : 10369085,
    "failures" : [ ],
    "shards" : {
      "total" : 330,
      "failed" : 0,
      "successful" : 330
    }
  }, {
    "snapshot" : "2019-02-01",
    "uuid" : "pHwGshbJRGO-C47uCuuFDw",
    "version_id" : 6040299,
    "version" : "6.4.2",
    "indices" : [ "logs", "awswaf-2019-02-13", "logstash-2019.02.06", "filebeat-6.6.0-2019.02.19", "logstash-2019.02.13", "awswaf-2019-02-21", "awswaf-2019-02-24", "logstash-2019.02.15", "logs-2019-02-15", "filebeat-6.6.0-2019.02.26", "logstash-2019.02.21", "logs-2019-02-13", "awswaf-2019-02-01", "filebeat-6.6.0-2019.02.27", "logstash-2019.02.20", "awswaf-2019-02-07", "awswaf-2019-02-26", "awswaf-2019-02-10", "kibana_sample_data_flights"],
    "include_global_state" : true,
    "state" : "IN_PROGRESS",
    "start_time" : "2019-02-27T06:51:30.836Z",
    "start_time_in_millis" : 1551250290836,
    "end_time" : "1970-01-01T00:00:00.000Z",
    "end_time_in_millis" : 0,
    "duration_in_millis" : -1551250290836,
    "failures" : [ ],
    "shards" : {
      "total" : 0,
      "failed" : 0,
      "successful" : 0
    }
  } ]

解決方案2
 0  2017-12-27 11:27:36

拍攝快照后，您可以刪除一次索引。 您始終可以使用“ _restore”還原每個索引。

查看以下鏈接，了解如何快照，還原和刪除索引。

http://www.datawrangler.in/2017/12/es-index-s3-snapshot-restoration.html

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 恢復自動 AWS elasticsearch 快照所需的權限 Elasticsearch 6.3 (AWS) 快照恢復進度錯誤：“/_recovery is not allowed” Elasticsearch 快照失敗 - 無法恢復 使用 ansible 從 aws 快照恢復實例 使用 Python BOTO3 在 AWS 中恢復 RDS 快照 AWS - Neptune使用SDK從快照還原 AWS Elasticsearch Service - 手動快照到 S3 AWS Elasticsearch 快照是否包含數據？ AWS Aurora：如何通過 aws cli 恢復數據庫集群快照？ 如何從另一個 AWS 賬戶的快照恢復 Redshift 集群
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM