Elasticsearch index replication to new server

Question

I am very new to Elasticsearch and I have a architectural question. I have a system where I store bouch of documents. Some of these documents are PUBLIC some of them are Private.

In the main system the documents are entered, indexed and searched. This is so far very easy to implement.

Although here is an additional requirement.

• I need to create a slave system which is a replication of the primary main ES instance although it can not contain any private document not even in the index level. So it is not enough for me to make sure I search for the right documents with a filter but the index can not contain the document which are marked private

I was wondering if I could do the following ->

1, Clone the primary ES instance 2, Delete all documents which are private 3, Re-index the whole index 4, Create a snapshot of the clean ES 5, Restore this snapshot to the Slave Public instance.

This sounds very complicated to me. is there a better / easier way to do this ?

Thanks a lot for any pointer in advance,

Answer 1

I'd suggest this approach as an alternative. Create two indexes, Public and Private, on your main Elasticsearch cluster. Also create an index alias http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html that covers both so that you have a convenient way to query both indexes at once.

You can then use the Elasticsearch snapshot and restore http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html to take a snapshot of just the Public index and restore it to another cluster.

This way you have another ES cluster with only the Public index in it.

Another approach would be to keep each index, Public and Private, in it's own cluster. When you want to run a query across both Public and Private use a tribe node. A tribe node will connect to both clusters and run the query across both of them: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html