How do I best handle this pagination with asynchronous Node.js?

Question

My question involves the use of asynchronous code.

Here's the relevant pseudocode:

function handleQueryMore(result) {
    conn.queryMore(result.nextRecordsUrl, function(err, resultMore){    
        if (err) {}
        //do stuff with result
        if (!resultMore.done){
            handleQueryMore(resultMore);
        }
    });
}

//(below code runs first when launched of course)

var conn = new jsforce.Connection({////etc
var sql = "SELECT //.......

conn.query(sql, function(err, result) {
    if (err) {}
    //do stuff with result

    if (!result.done) //didn't pull all records
    {
        handleQueryMore(result);
    }               
});

The initial query only returns a certain maximum # of records and then handleQueryMore() is called recursively to handle each additional chunk of records.

Originally I had a single conn.query() in a where (!result.done){} loop but of course the problem there was that the conn.query() code runs asynchronously and doesn't get a chance to finish before the next run of the loop (resulting in an ~infinite purposeless loop).

There are 3rd party library ways to get the code to run syncronously but my guess is that there is some fundamental design paradigm that I'm not following. The recursion works but I fear the amount of memory that could be needed if the query returns a huge # of records.

Even if I know the # of records I need to paginate through, I MUST have the result.nextRecordsUrl to paginate, which I can't get until each previous query is executed... so I can't just run them all at the same time.

Anyone care to weigh in on this?

Thanks!

Answer 1

Use async.js package.

function handleQueryMore(result) {

  conn.queryMore(result.nextRecordsUrl, function(err, resultMore) {
    if (err) {}
    //do stuff with result
    else {
      async.each(result.data, function(individualData, callback) {
        // do some async task
        callback(); // to tell that this is completed
        //callback(err); if there is any error while processing data
      }, function(error) {
        if (!error) {
          if (!result.done) //didn't pull all records
          {
            handleQueryMore(result);
          }
        }
      });
    }
  });


}



var conn = new jsforce.Connection({ ////etc
  var sql = "SELECT //.......

  conn.query(sql, function(err, result) {
    if (err) {} else {
      async.each(result.data, function(individualData, callback) {
        // do some async task
        callback(); // to tell that this is completed
        //callback(err); if there is any error while processing data
      }, function(error) {
        if (!error) {
          if (!result.done) //didn't pull all records
          {
            handleQueryMore(result);
          }
        }
      });
    }
  });
});

Answer 2

I wouldn't do it that that way at all. It is horribly inefficient as you get bigger data sets. Instead I'd take in querystring parameters that indicate the record index I want to start at and the count to return. All databases have an equivalent of skip and take (eg if you're using the jsforce method chain API you can do .skip(5).take(10) to return records 6 - 16 in a set). This simplifies your code a lot and reduces risk by making each request independent of others.

Also, I notice the comment //(below code runs first when launched of course) , but that's also not a great idea. You ideally want to query the db on demand, not on app start. That also might be why you're running into the challenges that you're having.

Answer 3

Hii if u are using mongo db then try this..

   user.aggregate({ $match: { status: true, type: req.query.type } },
        { "$skip": (skipValue) }, { "$limit": limitValue },
        (err, data) => {
//Code
})