Can we use Elastic Search as Backend Data Store by replacing Postgres?

Question

We are using Postgres to store and work with app data, the app data contains mainly:

1. We need to store the incoming request json after processing that.
2. We need to search the particular JSON using the Identifier field, for which we are creating a separate column, for each row in the table
3. For Clients, they may require searching the JSON column, I mean client want to one json based on certain key value in the json

All these things are ok at present with Postgres, when I am reading some blog article, where they mentioned that we can use ElasticSearch as backend data store also, instead of just as search server, if we can use like that, can we replace Postgres with ElasticSearch? What advantages I can get in doing this, what are the pros of postgres when compared with ElasticSearch for my case, what are cons?

Can anyone given some advice please.

Answer 1

Responding the questions one by one:

1. We need to store the incoming request json after processing that.

Yes and No. ElasticSearch allows to store JSON objects . This works if the JSON structure is known beforehand and/or is stable (ie the same keys in the JSON have the same type always).

By default the mapping (ie schema of the collection) is dynamic , means it allows to infer schema based on the value inserted. Say we insert this document:

{"amount": 1.5}  <-- insert succeeds

And immediately after try to insert this one:

{"amount": {"value" 1.5, "currency": "EUR"]} <-- insert fails

ES will reply with an error message:

Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@757a68a8; line: 1, column: 13]

If you have JSON objects of unknown structure you can still store them in ES, it could be done by using the type object and setting property enabled: false ; this will not allow you to do any kind of queries on the content of such field though.

2. We need to search the particular JSON using the Identifier field , for which we are creating a separate column ,for each row in the table

Yes. This can be done using field of type keyword if identifier is an arbitrary string, or integer if it is an integer.

3. For Clients, they may require searching the JSON column, I mean client want to one json based on certain key value in the json.

As per 1), yes and no. If JSON schema is known and strict, it can be done. If JSON structure is arbitrary, it can be stored but will not be queryable.

Though I would say ElasticSearch is not suitable for your case, there are some some guys that make JDBC and ODBC drivers for ElasticSearch, apparently in some cases ElasticSearch can be used as relational database.

Answer 2

elasticsearch is a HTTP wrapper to Apache Lucene. Apache Lucene stores object in a columnar fashion in order to speed-up search (Lucene segments).

I am completing the very good Nikolay answer:

The good:

• Both Lucene and Elasticsearch are solid project
• Elasticsearch is (my opinion) the best and easiest software for clustering (sharding and replication)
• Support version conflict ( https://www.elastic.co/guide/en/elasticsearch/guide/current/concurrency-solutions.html )

The bad:

• not realtime ( https://www.elastic.co/guide/en/elasticsearch/guide/current/near-real-time.html )
• No support ACID transaction (Changes to individual documents are ACIDic, but not changes involving multiple documents.)
• Slow to get lot of data (must use search scroll, very slow comparing to a SQL database fetch)
• No authentication and access-control

My opinion is use elasticsearch as a kind of view of your database, with read-only access.