MongoDb Command Failed Get More Mongo C# Driver

Question

This issue is not easy to reproduce unless you have a lot of data.

I have a lot of queries that are aggregates that return massive amounts of data (tens of millions of results). The results of these queries are obviously too large to store in memory so when I return the results I do a ToEnumerable() . When I iterate over the returned IEnumerable I eventually get an error

MongoDB.Driver.MongoCommandException: Command getMore failed: cursor id xxxxxxxxxx not found

I am using the latest C# driver (2.6.1)

        var agg = collection.Aggregate()
            .Group(new BsonDocument
            {
                { "_id" , "$FileID" },
                { "id", new BsonDocument
                    {
                        {  "$min", "$_id" }
                    }
                },
                { "Count", new BsonDocument
                {
                    {"$sum", 1 }
                } }
            })
            .Match(new BsonDocument
                { { "Count", new BsonDocument { { "$gt", 1 } } }
            });

        agg.Options.AllowDiskUse = true;

        var results = agg.ToEnumerable();
        var deletedCount = 0;

        // exception occurs here

        foreach (var result in results)
        {
            collection.DeleteOne(x => x.Id == result["id"].AsObjectId);
            deletedCount += 1;
            Console.WriteLine("Deleted Count: " + deletedCount);
        }

I researched this issue and it sounds like the issue is that the cursor is timing out, but I have tried removing the cursor timeout with no luck. The only way I was able to get around this issue was to use a ToList() instead, which loads the entire results into memory which I don't want to do with these large queries.

Any help would be appreciated, I have spent a lot of time banging my head against this with little to no results.

Answer 1

This actually reminds me of an issue I was once having with a server-side cursor and it might well be a similar problem here. You are modifying ( .DeleteOne ...) the collection that your are iterating over. In my case, I was having a simply query that I was iterating over and in that loop I would insert documents into MongoDB which would get picked up by the cursor. That gave me an endless loop...

Now, you are deleting things here but, still, that could cause some funky behaviour, I suppose. Would it perhaps be an option a) to try using a separate MongoDB client for the .Delete() ? I doubt this helps at all but it might be worth a try b) instead of deleting straight away simply keeping a list of, well, tens of millions of ObjectId s before moving on to a bulk deletion ( DeleteMany() ) which would probably speed up things anyway compared to deleting individual documents?