Node.js Rendering big amount of JSON data from the server

Question

I have an view that has a for loop that inserts rows to a table. The table is very big and already consisting of couple of thousand of rows.

When I run it, the server throws out of memory exception.

I would like to add an infinite scrolling feature so I won't have to load all the data at once.

Right now the data is being sent with regular res.render(index.ejs, data) ( data is JSON)

I can figure out the infinite scrolling part, but how do I get the JSON data in chunks from the server ?

I am using node.js with express and ejs as template engine. I am open for using any framework that will aid me through the process (was particularly checking out Angualr.js).

Thanks

Answer 1

Firstly, there is an angular component for infinite scroll: http://ngmodules.org/modules/ngInfiniteScroll

Then, you have to change you backend query to look something like:

http://my.domain.com/items?before=1392382383&count=50

This essentially tells your server to fetch items created/published/changed/ whatever before the given timestamp and return only 50 of them. That means you back-end entities (be them blog entries, products etc) need to have some short of natural ordering in a continuous space (a publication date timestamp is almost continuous). This is very important, cause even if you use a timestamp, you may end-up with extreme heisenbugs where items are rendered twice (if you use <= that's definate), loose entries (if you use < and 2 entries on the edges of your result sets are on the same timestamp) or even load the same items again and again (more than 50 items on the same timestamp). You have to take care of such cases by filtering duplicates.

Your server-side code translates this into a query like (DB2 SQL of course):

SELECT * FROM ITEMS
WHERE PUBLICATION_DATE <= 1392382383
ORDER BY PUBLICATION_DATE DESC
FETCH FIRST 50 ROWS ONLY

When infinite scroll reaches the end of the page and calls your registered callback, you create this $http.get request by taking into account the last item of your already loaded items. For the first query, you can use the current timestamp.

Another approach is to simply send the id of the last item, like:

http://my.domain.com/items?after_item=1232&count=50

and let the server decide what to do. I suppose you can use NoSQL storage like Redis to answer this kind of query very fast and without side-effects.

That's the general idea. I hope it helps.