I am trying to get ALL documents from a collection in my Cosmos DB I have on Azure. The collection contains approx 50.000 documents.
I get this error: MongoError: cursor does not exist, was killed or timed out
when I am doing this:
const mongoose = require('mongoose');
const mongooseOptions = { useNewUrlParser: true };
mongoose.connect(connectionString, mongooseOptions);
mongoose.set('useCreateIndex', true);
mongoose.Promise = global.Promise;
const mongoDB = mongoose.connection;
mongoDB.on('error', console.error.bind(console, 'MongoDB connection error:'));
const Schema = mongoose.Schema;
const MongoEidModelSchema = new Schema({
uid: { type: String, unique: true },
eid: { type: String, unique: true }
});
const MongoEidModel = mongoose.model('eids', MongoEidModelSchema);
MongoEidModel.find({}, {timeout: false}).then(data => {
console.log(data);
console.log(Object.keys(data).length);
});
When I set a limit of 1000 or 1500 on the find()
it works.
I have also tested to change the RU/s on the collection from 400 to 10.000 (in the Azure Portal / console) which also works, but that seems like an expensive solution... doesn't it?
I have also tested to fetch this with find()
in batches in a recursive loop until there is no more documents left, with a sleep between each iteration (otherwise Cosmos DB gives me "429: Too many requests" after a while.
Is there a way in which I can get ALL the 50.000 documents using Node.js and Mongoose without changing RU/s or doing recursive loops?
Thanks in advance!
/Daniel
To avoid confusion, I assume you're using the MongoDB driver to access Cosmos in Azure?
For MongoDB, there is a query limit of 16Mb (which you may well be shooting past if you are returning 50k documents). See here: https://docs.mongodb.com/manual/reference/limits/
It is possible that the limitation isn't enforced in the node driver (I haven't inspected its source), in which case it's worth consulting the Azure docs: https://docs.microsoft.com/en-us/azure/cosmos-db/faq
The upshot is, you should really use a cursor to walk across the collection when you are dealing with large numbers of documents like this. See here: How can I use a cursor.forEach() in MongoDB using Node.js?
Hope this helps :)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.