Out of Memory Exception Entity Framework When Saving

Question

I'm looping through many different objects in entity framework, grabbing some related data and saving that data. I'm opening a new thread for each item since it takes a while to do the work in the method. However, I'm getting out of memory exceptions a couple thousand items into the loop. I'm trying to figure out where this problem is occurring and how to keep the multi-thread functionality, but fix the error. Any help is greatly appreciated. Thank you.

    foreach (var prod in products)
    {
        try
        {
            new Thread(() =>
            {
                ParsePage(prod);
            }).Start();


            counter++;
            Console.WriteLine(counter);
        }
        catch (Exception ex)
        {
            string message = ex.Message;
        }
    }

    public void ParsePage(Product p)
    {
        //New Repository
        IDataRepository repo = new DataRepository();

        //Grab related page report
        PageReport pr = repo.PageReports.Where(c => c.ThePageType == "Product").FirstOrDefault(c => c.ThePageTypeID == p.ProductID);

        //Do some stuff

        //Save
        repo.SavePageReport(pr);

    }

Answer 1

You may be running out of memory because each thread has a default stack size of 1 MB . 2000 started threads implies 2 GB of stack, which may be the max on your system :

Memory Limits for Applications on Windows*

32-bit

• Static data - 2GB
• Dynamic data - 2GB
• Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

Note that on 32-bit Windows, the sum of all types of data must be 2GB or less. The practical limit is about 1.75GB due to space used by Windows itself

64-bit

• Static data - 2GB
• Dynamic data - 8TB
• Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

See also this article by Raymond Chen and this stackoverflow question for more information.

As KroaX said, the newer task and older threadpool libraries are designed to handle situations like this. They offer an additional level of abstraction between threads and tasks, allowing you to easily create thousands of lightweight tasks that will be assigned to threads as they become available.

Answer 2

If you're using the same DbContext for the entire operation, its cache may cause the Out of memory error. I had a similar situation when parsing and storing large amounts of data from XML files.

You should make your repository disposable, as dbc suggests, and then wrap it in a using clause:

public void ParsePage(Product p)
{
    using (var repo = new DataRepository())
    {
        //Grab related page report
        PageReport pr = repo.PageReports.Where(c => c.ThePageType == "Product").FirstOrDefault(c => c.ThePageTypeID == p.ProductID);

        //Do some stuff

        //Save
        repo.SavePageReport(pr);
    }
}

Info on how to implement the Disposable pattern can be found here