Elastic search API Vs Spring data Vs logstash

Question

I am planing to use elastic search for our dashboard using spring boot based rest services. After research i see top 3 options

Option A:

1. Use Elastic Search Java API ( from comment looks like going to go away)
2. Use Elastic Search Java Rest Client
3. Use spring-data-elasticsearch ( planing to use es 5.6 but challenging for latest es 6 as I don't see it's supports right now)

Option B:

Or shall I use logstash approach to Sync data between postgressql and elastic search using logstash ?

Which one among them will be long term approach to get near real time data from ES in high load scenario ??

Usecase: I need to save some data from postgresql table to elastic search for my dashboard (near real time )

Update is frequent for both tables and es to maintain current state

Load is going to increase in couple of week

Answer 1

The options you listed, in essence, are: should you go with a ready to use solution (logstash) or should you implement your own.

Try logstash first to see if it works for you - it'll take less time than implementing your own solution, and you can get working solution in minutes (if it's not hundreds of tables)

If you want near-real time, then you need to figure out if it allows you to:

• handle incremental updates, ie if its 'tracking_column' configuration will work for your data structure and it will only load updated records in each run, not the whole table.
• run it at the desired frequency
• and in general, satisfies your latency requirements

If you decide to go with your own solution, keep in mind that spring-data-elasticsearch is a higher level wrapper for underlying elasticsearch client. If there are latency goals, then working on the lower level (elasticsearch clients) may give you better control and more options to tune the pipeline.

Otherwise, the client choice will not matter that much as data feed features (volume/update frequency) and db/es cluster configuration.