MongoDB NodeJS本机游标过早关闭

Question

I'm executing a process to each element of a ursor, my collection has around 6 million documents, the entire process would take up to 10 hours since I have to process a profile at a time and it's a somewhat complex process for each one.

 var cursor = dbMain.collection("profiles").find({});

 var getNext = function(){
      cursor.nextObject(processOne);
 };

 var processOne = function(err, profile){
      if(err){
           console.error("Error loading profile", err);
           getNext();
      } else if(profile == null){
           // process is done, all profiles processed!!!
      } else {
           processProfile(profile)
               .then(function(){
                    getNext();
               })
               .catch(function(errProfile){
                    console.error("Error processing profile", errProfile);
                    getNext();
               });
      }
 };

 var processProfile = function(profile){
      // Complex profile processing and promise resolve when done
 };

 getNext();

PROBLEM: I get null (which would mean the cursor has been exhausted) when i'm around 300,000 profiles, so, only the first 300,000 or so get processed. ¿Anyone knows how to deal with it or the reason I'm getting null way before time?

Answer 1

It seems I've found the reason:

Since it's a cursor that needs to be alive a long time, in the find options I had to add "timeout : false".

 var cursor = dbMain.collection("profiles").find({}, {timeout : false});

In the driver documentation says it's false by default (well it's not!!) , now all of my profiles get processed.

Answer 2

For anyone who stumbles upon this one on latest MongoDB drivers, be aware of the following. If you call .hasNext() twice after the last .next() call, you will get: Error: Cursor is closed .

For example:

while(await cursor.hasNext()) {
    await cursor.next();
}

if (await cursor.hasNext()) {} // Error!

if (!cursor.isClosed() && await cursor.hasNext()) {} // Correct way

Answer 3

I had the same issue, i resolved it by pushing each document of the cursor on a list and then returning the list, as shown here:

const list:any= [];
const cursor= await client.db('insertYourDb').collection('insertYourCollection').find().forEach(function(document) { 
        list.push(document) 
    });
return list;