Elasticsearch index Architecture for large data and having more update/delete operations

Question

I have a index which has almost now 50GB of data and it will exceed to 100GB to soon! so now I would like to setup index architecture for the better performance.

I have checkout many things one of them is Index LifeCycle but as the index which i have that can be updated at anytime! so in that case how can I design my index so that will be good for perfomance.

Another thing is that As I have found an artical Dynamic Indices on the update and delete records from Index. it shows that data will be find out while we perform any search operations! as in my case I have too much update records in those cases it will reduce the performance of the index!

How can we improve our index performance when we have a large data and having to many update and delete operaion? what architecture should we follow?

Answer 1

Is all of your date likely to be updated or deleted or is it only the latest data? If your updates are on fairly recent data and the old data is read-only you can create the hot-warm-cold architecture as described in this blog post.

If all of your data is likely to be updated you can do a hot-warm architecture, where all your updates go to hot node and all queries go to warm node. Elastic will sync the hot and warm node to achieve eventual consistency so you might have to live with stale date for milliseconds I assume. Check this .

In my experience Elastic is able to easily handle 50-100 GB data even if you update and search from the same set of indices and nodes. It all depends on the rate of the updates and search.