Mongo DB Cursor issue while iterating over a huge collection

Question

I am iterating over a Mongo DB collection. This collection has millions of documents in it. I am using java API for this iteration. This iteration exercise can go on for a while and we dont want to impact performance of the application. So we are planning to use Thread.sleep during this iteration. But we are facing cursor timeout kind of problems. Anyone faced this before? Also, is it possible to read data in chunks eg 100 documents at a time?

Answer 1

Instead of using a cursor over the entire collection you can try paging through the collection by the _id. So each time query for 100 documents (order by _id) and keep the last _id you encounter. Then on each consecutive query use a condition to fetch documents where _id > last _id from previous fetch.