Where to store data for a search index?

Question

I'm building a search index (Elasticsearch) for existing data in a relational database. Is it common to have duplicated in both the relational database as well as the search index? Or should it just be the search index? How safe is it to have your data stored entirely on say the Elasticsearch cloud?

Answer 1

It's a common pattern to have data in both systems:

• datastore: source of truth. Only used for CRUD
• search engine: Where all queries are sent

If you can afford some data loss (depending on your use case) in very corner cases, then having only data in elasticsearch could be ok.

I recommend reading carefully this page: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html

Answer 2

I Would recommend keeping both the RDBMS as a storage and Index file of Elastic Search to have data.

You can run reindexing of the Elastic search every day at midnight to keep the data up to date.

If the Index file corrupts, you still have data in RDBMS, which can be used to rebuild the index later.