How to show huge amount of report data client side (mongoDB, JS)

Question

Due to the nature of the data that we collect in our application, its necessary to allow user to be able to search through 1,000,000+ rows of data, and receive results client side as fast as possible.

We have played around with various techniques, and currently experimenting with the document storage engine MongoDB. We are essentially storing the report data in Mongo in the way we want to show it on the client side.

But, this is where i wanted to lean on your experience. We have used some report table plugins such as datatables.net, but when a client requests data which may match 100,000+ records, the latency in delivering that data to the client side, can take well over a minute to export the JSON from server to client.

I have tried filtering the resultset by only sending 1000 result at a time to the client, but what if the user decides to sort a column? The entire request has to be performed again, and the 1st 1000 results loaded into the client.

But what if the client chooses to search within a column, the results have to be re-fetched again. All of these problems dont exist when all of the data has been loaded into the client side.

So essentially my question is, how can you efficiently and speedily, allow clients to search and manipulate large datasets, but have the results delivered to the client side in a manner which doesnt require every result to be sent through in one go, therefore keeping the client side as lightweight as possible?

Im sure there must be some front-end datatable type plugins for MongoDB, potentially via node.js for direct JSON communication, but i cant find anything!

Thanks for your help.

Answer 1

It sounds like your problem lies with the sheer volume of data you are trying to pass client-side. What screen displays 1,000 rows of data??

Make a request with a start index, a sort order and a page size and fetch only that much data in that order and return it client-side.

Take a look at jqGrid for example as a client-side grid that passes paging and sorting information back to the server and only retrieves one page-worth at a time. jqGrid with MongoDB and the correct indexes in place is blindingly fast.

Answer 2

MongoDB is great, however you may want to also index this content outside of MongoDB to provide the performance and flexibility you describe. Take a look at Apache Solr or Elastic Search . Both of these are based on Lucene, which has very robust searching capabilities. In particular you can execute a query with paging and sorting in a very performant manner. Both of these tools can return the resultset in JSON. With that many records, and using a something based on Lucene you will also be able to provide more advanced filtering capabilities very easily such as facted browsing and stemming.