How do i access a large amount of data fastly from Mongodb when the results from the query need to be manipulated before sending to the client

Question

I have a Mongodb database with React based front-end and Node based back-end with approximately 100000 documents in the collection. I have already done indexing and I am already using Redis. I have implemented a search functionality to find all the relevant documents with 'title' property containing the search query and display them to the user. I am also using pagination displaying 20 results per page and there are various filters which a user can apply on the result.

Till now what I have done is when a user searches something, a text search on the database is performed. All the results of the search are then scored based on their relevance depending on the search query and they are sorted in the order of the scores and returned to the client. Then when the page is changed or a filter is applied, the changes take place on the javascript front-end side only. Approximately 9000 results means approximately 11 MB of data which takes a hefty amount of time to be sent from the back-end.

One thing I could do is for every search retrieve all the data from the database and score them and sort them based on scores and return only 20 results to be displayed on the first page but then every time the page is changed or a filter is applied, I will have to do the complete thing again which will take a long time for something as simple as filtering.

I would like to know how do I go about it to get the data faster on the client-side while not affecting the speed of filtering and changing pages by a lot.

A suggestion of additional or a different tech stack is also welcome. Thank you.

Answer 1

Looks like all this could be easily solved using elasticsearch. Don't do anything on the client side. All complex queries can be easily handled by elasticsearch in a fraction of milli second.