How can I use elastic search indexes exclusively "in-memory"?

Question

I have a class that stores documents and indexes them with an elasticsearch client. I'd like for indexes to be in-memory only unless explicitly saved to disk. Specifically:

• can create an index and search it
• is wiped from disk when session ends if unsaved
• if saved, is persisted to disk

However, Elasticsearch is effectively an on-disk service (writes index directly to disk, removes when asked).

Is there a way to get the desired behavior from Elasticsearch?

Answer 1

That feature was removed in a long time ago (in ES 1.7). It was possible to leverage the memory store to store everything in memory and not on file.

This was deprecated and removed in ES 2.0 for obvious reasons, ie that didn't offer any resiliency in case a node would break.

Something similar is the ability to preload everything in memory when starting up, even though that doesn't completely fulfill your requirement. Feel free to expand on why you'd require this kind of behavior.

Answer 2

You're correct that Elasticsearch documents are saved in the disk, But the data is cached in RAM so that our /GET queries will get a faster response while search as long as you've enough memory.

you can explicitly mark the indexes store type as cached using the below query to store the shard index on the file system (maps to Lucene MMapDirectory) by mapping a file into memory

curl -XPUT /indexname { "settings": { "index.store.type": "mmapfs" } }

mmapfs

The MMap FS type stores the shard index on the file system (maps to Lucene MMapDirectory) by mapping a file into memory (mmap). Memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure you have allowed plenty of virtual address space.

Read more about the index store type here: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-store.html

Also if you're dealing with huge data on the Elasticsearch and recent days indexes are queried more frequently you can try ElasticSearch Hot-warm node architecture and for the hot node, you can apply the index store settings to mmapfs possibly for hot nodes alone

More Info: https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x