C#: Releasing memory usage

Question

I need to execute a long-heavy process for a question out of the box. So I have divided the process in multiple subprocess. The question now, is how to release memory, before the execution of each window.

It's easier to explain with an example. Let's take a look to this pseudo-code.

1. Some earlier code to do other things
2. Do
3.     Raise a Task
4.     If raised-task > 1000
5.         Wait all raised task to finish
6.         Release Memory
7.     End If
8. Wile Something not relevant

With that idea, I have developed the next method, which gets executed every time it's reached the thread limitation:

List<Task> LTask();
//It's not relevant, but this list is populate like
//var task = Task.Run(() => something());
//LTask.Add(task);

private void waitForAll()
{
    //Break point 1
    Task.WhenAll(LTasks).Wait();

    LTasks.Clear();
    LTasks = null;
    GC.Collect();
    GC.WaitForPendingFinalizers();

    //Break point 2
    LTasks = new List<Task>();
}

I expected memory gets constant (whit some variation) around some values. I mean:

1. Thread limit is reached
2. Snapshot of memory usage with visual studio diagnosis tools in BreakPoint 1 --> 100MB
3. Snapshot of memory usage with visual studio diagnosis tools in BreakPont 2 --> 100 MB. First question, why this has not decreased? All the threads are finished and I forced Garbage Collector to execute.

Next time the limit is reached and this code is executed, if I take a snapshot again, the memory keeps increasing: 200, 300, ...

This is a capture of diagnosis tools. Odd snapshots are taken every time break point 1 is reached, and Even snapshots on break point 2.

Second question, this will continue increasing whit no limit until it throws an Out of memory Exception ?

Last question, any alternatives to solve the problem, and release the memory?

UPDATE 1: After some tests, and thanks to the comments, I have developed a test code, to dig into it. There has to be involved something else. Plase, take a look to the next piece of code. The memory continue increasing with no limit.

private List<Task> LTasks = new List<Task>();
private void manageThreadholdLimit()
{
    waitForAll();
    realeaseMemory();
}

private void waitForAll()
{
    Task.WhenAll(LTasks).Wait(); 
    LTasks.Clear();
    LTasks = null;  
}
private void realeaseMemory()
{   
    GC.Collect();
    GC.WaitForPendingFinalizers();

    LTasks = new List<Task>();
}
public void Main(){
    int i = 0;

    while (true)
    {
        i++;

        var task = Task.Run(() => Thread.Sleep(100));
        LTasks.Add(task);

        //Si hemos alcanzado el máximo de paralelismo, esperamos la conclusión
        if (i % 1000 == 0) manageThreadholdLimit();

    }
}

Answer 1

GC collection is slightly different in debug, see:( John Skeet knows all ) So I would do allot of logging when running this in release mode to verify all behaviours.

Any solution will be very dependent on your actual code and if there are unmanaged resources being accessed.

That said I have had to deal with a problem like this before and I have "solved" it in 2 different ways before.

One solution

Have counters that are incremented in constructor and decremented in the finalizer in the class that does the actual work and wait for the counter to fall under a defined threshold and importantly run collect again for the finalized objects to be collected.

Be careful checking the total memory consumption before continuing otherwise you can end up with out of memory exception.

Now this will actually increase your memory consumption slightly. For more info see

Another solution

Have a loop waiting for the memory consumption to fall by using GC.GetTotalMemory() or even better performance counters and wait for it to come down.

This can end up not doing any work at all if your resources are not being collected.

Answer 2

It is not guaranteed that the GarbageCollector will run immediately after you unreference the object. In fact, it probably won't. Even if you want to call it manually for testing purposes with GC.Collect() you have no real guarantees it will run immediately. Also, there is a cost to calling GC often. RAM is there to be used (at least in newer machines with lots of RAM...). You have a problem if your memory goes up, and stays up after a longer while. This usually indicates that you have some other memory problem. Maybe you have a leak?

If we're talking about free solutions, you can use ProcessExplorer and CLR Profiler to look for potential memory problems. Here is an article on how to do that.

Things to look out for:

1. Set references to your objects to null as soon as you can. You are done with the list? Set its value to null .
2. Split the method that is working with a lot of data into smaller methods - even if you set something to null , the GC won't run until a method is exited.
3. Check if you have finalizers properly implemented where needed.
4. Make sure you do not have leaks: check if the objects you are working with aren't referenced from outside the worker method. Be especially careful about event handlers and data bindings.

---

Also, Task.WhenAll() apparently keeps the references to all its "child" tasks. If you manually call GC immediately after it in the same method, I think there is a chance it won't be able to touch that memory, as the method itself is still "referencing" it.

Here is a MSDN article about garbage collection.

Answer 3

I think I found an explanation, thanks to @ScottChamberlain's answer at: Why Garbage Collector doesn't collect Tasks objects

I think the memory gets increasing because TaskScheduler is static, and still stores the task's reference, despite the task is finished. So the amount of references is which keeps increasing the memory, not the memory usage of each task itself.

I have not figure out yet how to remove that references, but, as an alternative solution, I could refactorize the code the use threads. For example, this code is an alternative of the "Update 1" in my question. And the memory remains stable on 18mb.

private static List<Thread> LThreads = new List<Thread>();
private static void manageThreadholdLimit()
{
    waitForAll();
    realeaseMemory();
}

private static void waitForAll()
{
    LThreads.ForEach(x => x.Join());
    LThreads.ForEach(x => x = null);
    LThreads.Clear();
    LThreads = null;
}
private static void realeaseMemory()
{


    GC.Collect();
    GC.WaitForPendingFinalizers();

    LTasks = new List<Task>();
    LThreads = new List<Thread>();
}
public static void Main(string[] args)
{
    int i = 0;

    while (true)
    {
        i++;

        var t = new Thread(() => Thread.Sleep(100));
        LThreads.Add(t);
        t.Start();

        if (i % 1000 == 0) manageThreadholdLimit();

    }
}