Using Lucene as storage

Question

I would like to know if it would be recommended to use Lucene as data storage. I am saying 'recommended' because I already know that it's possible.

I am asking this question because the only Q&A I could find on SO was this one: Lucene as data store which is kind of outdated (from 2010) even if it is almost exactly the same question.

My main concern about having data exclusively in Lucene is the storage reliability. I have been using Lucene since 2011 and at that time (version 2.4) it was not improbable to encounter a CorruptIndexException, basically meaning that the data would be lost if you didn't have it somewhere else. However, in the newest versions (from 4.x onward), I've never experienced any problem with Lucene indices.

The answer should not consider the performance too much as I already have a pretty good idea of what to expect in that field.

I am also open to hear about SOLR and ElasticSearch reliability experiences... (how often are the shards failing, what options do we have when this occurs, etc)

Answer 1

This sounds like a good good match for Solrcloud as it is able and willing to handle the load and also takes care of the backup. My only concern would be that it is not a datastore, it "only" works with the indexing of those documents.

Answer 2

We are using SolrCloud for data storage and reliability is pretty good till now. However make sure that you configure and tune it well or else you could find nodes failing and zookeeper being unable to detect some of them after some time..