How to fetch/count millions of records in mongodb with nodejs

Question

We have a collection with millions of records in mongoDB. its taking lots of time and time out to count and create pagination with these records. whats the best way to do it using nodejs. I want to create a page where I see records with pagination, count, delete, search of records. Below is the code which doing query to Mongo with different conditions.

crowdResult.find({ "auditId":args.audit_id,"isDeleted":false})
            .skip(args.skip)
            .limit(args.limit)
            .exec(function (err, data) {
                if (err) 
                    return callback(err,null);
                console.log(data);
                return callback(null,data);
            })

Answer 1

If the goal is to get through a large dataset without timing out then I use the following approach to get pages one after another and process the paged resultset as soon as it becomes available: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f

Please focus on the following lines to get a quick idea of what the code is doing before diving deeper:

1. Let getPage() handle the work, you can set the pageSize and query to your liking: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L68
2. Method signature: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L29
3. Process pagedResults as soon as they become available: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L49
4. Move on to the next page: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L53
5. The code will stop when there is no more data left: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L41
6. Or it will stop when working on the last page of data: https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L46

I hope this offers some inspiration, even if its not an exact solution for your needs.