使用存储库模式时的 MongoDB 和大型数据集

Question

Okay so at work we are developing a system using MVC C# & MongoDB. When first developing we decided it would probably be a good idea to follow the Repository pattern (what a pain in the ass!), here is the code to give an idea of what is currently implemented.

The MongoRepository class:

public class MongoRepository { }

public class MongoRepository<T> : MongoRepository, IRepository<T>
where T : IEntity
{
    private MongoClient _client;
    private IMongoDatabase _database;
    private IMongoCollection<T> _collection;

    public string StoreName {
        get {
                return typeof(T).Name;
            }
        }
    }

    public MongoRepository() {

        _client = new MongoClient(ConfigurationManager.AppSettings["MongoDatabaseURL"]);
        _database = _client.GetDatabase(ConfigurationManager.AppSettings["MongoDatabaseName"]);

        /* misc code here */

        Init();
    }

    public void Init() {
        _collection = _database.GetCollection<T>(StoreName);
    }

    public IQueryable<T> SearchFor() {
        return _collection.AsQueryable<T>();
    }
}

The IRepository interface class:

public interface IRepository { }

public interface IRepository<T> : IRepository
where T : IEntity
{

    string StoreNamePrepend { get; set; }

    string StoreNameAppend { get; set; }

    IQueryable<T> SearchFor();

    /* misc code */

}

The repository is then instantiated using Ninject but without that it would look something like this (just to make this a simpler example):

MongoRepository<Client> clientCol = new MongoRepository<Client>();

Here is the code used for the search pages which is used to feed into a controller action which outputs JSON for a table with DataTables to read. Please note that the following uses DynamicLinq so that the linq can be built from string input:

tmpFinalList = clientCol
    .SearchFor()
    .OrderBy(tmpOrder) // tmpOrder = "ClientDescription DESC"
    .Skip(Start) // Start = 99900
    .Take(PageLength) // PageLength = 10
    .ToList();

Now the problem is that if the collection has a lot of records (99,905 to be exact) everything works fine if the data in a field isn't very large for example our Key field is a 5 character fixed length string and I can Skip and Take fine using this query. However if it is something like ClientDescription can be much longer I can 'Sort' fine and 'Take' fine from the front of the query (ie Page 1) however when I page to the end with Skip = 99900 & Take = 10 it gives the following memory error:

An exception of type 'MongoDB.Driver.MongoCommandException' occurred in MongoDB.Driver.dll but was not handled in user code

Additional information: Command aggregate failed: exception: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in..

Okay so that is easy to understand I guess. I have had a look online and mostly everything that is suggested is to use Aggregation and "allowDiskUse:true" however since I use IQueryable in IRepository I cannot start using IAggregateFluent<> because you would then need to expose MongoDB related classes to IRepository which would go against IoC principals.

Is there any way to force IQueryable to use this or does anyone know of a way for me to access IAggregateFluent without going against IoC principals?

One thing of interest to me is why the sort works for page 1 (Start = 0, Take = 10) but then fails when I search to the end ... surely everything must be sorted for me to be able to get the items in order for Page 1 but shouldn't (Start = 99900, Take = 10) just need the same amount of 'sorting' and MongoDB should just send me the last 5 or so records. Why doesn't this error happen when both sorts are done?

---

ANSWER

Okay so with the help of @craig-wilson upgrading to the newest version of MongoDB C# drivers and changing the following in MongoRepository will fix the problem:

public IQueryable<T> SearchFor() {
    return _collection.AsQueryable<T>(new AggregateOptions { AllowDiskUse = true });
}

I was getting a System.MissingMethodException but this was caused by other copies of the MongoDB drivers needing updated as well.

Answer 1

When creating the IQueryable from an IMongoCollection, you can pass in the AggregateOptions which allow you to set AllowDiskUse.

https://github.com/mongodb/mongo-csharp-driver/blob/master/src/MongoDB.Driver/IMongoCollectionExtensions.cs#L53