DbContext OutOfMemoryException

Question

I have a DbContext with a dataset of >20M records, that has to be converted to a different data format. Therefore, I read the data into memory, perform some tasks and then dispose the DbContext. The code works fine, but after a while I get OutOfMemoryExceptions. I have been able to narrow it down to the following piece of code, where I retrieve 2M records, then release them and fetch them again. The first retrieval works just fine, the second one throws an exception.

// first call runs fine
using (var dbContext = new CustomDbContext())
{
    var list = dbContext.Items.Take(2000000).ToArray();
    foreach (var item in list)
    {
        // perform conversion tasks...
        item.Converted = true;
    }
}

// second call throws exception
using (var dbContext = new CustomDbContext())
{
    var list = dbContext.Items.Take(2000000).ToArray();
    foreach (var item in list)
    {
        // perform conversion tasks...
        item.Converted = true;
    }
}

Shouldn't the GC automatically release all memory allocated in the first using block, such that the second block should run as fine as the first one?

In my actual code, I do not retrieve 2 million records at once, but something between 0 and 30K in each iteration. However, after about 15 minutes, I run out of memory, although all objects should have been released.

Answer 1

I suspect you met LOH. Probably your objects are bigger than threashold and they are getting there, thus GC doesnt help by default.

Try this: https://www.simple-talk.com/dotnet/.net-framework/large-object-heap-compaction-should-you-use-it/

and see if your exception goes away.

ie add this between first and second part:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect(); 

Answer 2

IEnumerable has GetEnumerator() so you could try this to avoid .ToArray() or .ToList() that aren´t necessary if you just want to read:

// first call
using (var dbContext = new CustomDbContext())
{
    foreach (var item in dbContext.Items.Take(2000000))
    {
        // perform conversion tasks...
        item.Converted = true;
    }
}

// second call
using (var dbContext = new CustomDbContext())
{
    foreach (var item in dbContext.Items.Take(2000000))
    {
        // perform conversion tasks...
        item.Converted = true;
    }
}

Answer 3

Running GC will not help you, you have to run each iteration in different context. And dispose your context.

// ID is your primary key

long startID = 0;
while(true){
    using(var db = new CustomDbContext()){
        var slice = db.Items.Where(x=>x.ID > startID)
                            .OrderBy(x=>x.ID)
                            .Take(1000).ToList();

        // stop if there is nothing to process
        if(!slice.Any())
             break;

        foreach(var item in slice){
            // your logic...
            item.Converted = true;
        }

        startID = slice.Last().ID;
    }
}

If you want to process these things faster, alternate approach would be to run slices in parallel ....

Alternate Approach

I would recommend using dividing slices in 100x100, then I can process 100 slices of 100 items in parallel.

You can always easily customize slicing to meet your speed needs.

public IEnumerable<IEnumerable<T>> Slice(IEnumerable<T> src, int size){
     while(src.Any()){
         var s = src.Take(size);
         src = src.Skip(size);
         yield return s;
     }
}

long startID = 0;
while(true){
    using(var db = new CustomDbContext()){
        var src = db.Items.Where(x=>x.ID > startID)
                            .OrderBy(x=>x.ID)
                            .Take(10000).Select(x=>x.ID).ToList();

        // stop if there is nothing to process
        if(!src.Any())
             break;

        Parallel.ForEach(src.Slice(100), slice => {

             using(var sdb = new CustomDbContext()){
                 foreach(var item in sdb.Items.Where(x=> slice.Contains(x.ID)){
                     item.Converted = true;
                 }
             }

        } );

        startID = src.Last();
    }
}

Answer 4

After refactoring, memory gets released. I don't know why, but it works.

private static void Debug()
{
    var iteration = 0;
    while(true)
    {
        Console.WriteLine("Iteration {0}", iteration++);
        Convert();
    }
}

private static void Convert()
{
    using (var dbContext = new CustomDbContext(args[0]))
    {
        var list = dbContext.Items.Take(2000000).ToList();
        foreach (var item in list)
        {
            item.Converted = true;
        }
    }
}

When I move the content of Convert() to the while loop in Debug(), the OutOfMemoryExceptions is thrown.

private static void Debug()
{
    var iteration = 0;
    while(true)
    {
        Console.WriteLine("Iteration {0}", iteration++);
        using (var dbContext = new CustomDbContext(args[0]))
        {
            // OutOfMemoryException in second iteration
            var list = dbContext.Items.Take(2000000).ToList(); 
            foreach (var item in list)
            {
                item.Converted = true;
            }
        }
    }
}