ElasticSearch Snapshot across cluster

Question

Recently I created a snapshot of an index in an elasticsearch cluster consisting of 3 data nodes.

My main purpose is to move this index in another cluster.

Commands I used:

PUT /_snapshot/my_backup

{
    "type": "fs",
    "settings": {
        "location": "/home/me/backup",
        "compress": true
    }
}


PUT _snapshot/my_backup/snapshot_1?wait_for_completion=true
{
  "indices": "daily-20141127"
}

Note: Above queries were executed in the same data node.

Snapshot was successfully created in the above location but in another node from where I hit the queries and each node in this backup location had a folder(name:indices) which contained the shards of backup index on equivalent node.

Example:
node1: home/me/my_backup/indices/1,2,3
node2: home/me/my_backup/indices/4,5,6
node3: home/me/my_backup/indices/7,8,9

So I had to manually copy all indices folders across cluster to the node which had the metadata of snapshot to complete backup. I then copied backup folder to new cluster and restored it(restore procedure was fine)

Can all indices folders of backup index be in 1 node instead of being split across cluster?

Answer 1

The location you are specifying should be a "shared" location and the path specified in the repository should point to the same directory shared across all nodes. It cannot be just a local directory on the node. For example: that location can be a network-shared mount on a separate machine and all nodes should point to it and should be able to access it.

The reason for this is that each node may hold a primary or a replica from a shard and it should be able to make a copy of the docs contained in that specific shard and then moved over to the "shared" location.

Answer 2

As Andrei wrote - to use snapshots over multiple nodes, you will need a shared location which can be accessed by all nodes and you have to elasticsearch.yml settings with identical path.repo entry.

There is a guide how to create shared repositories across multiple clusters which covers all the steps you need, including setting up Samba server and clients + configuring ElasticSearch.