How does that thread cause a memory leak?

Question

One of our programs suffered from a severe memory leak: its process memory rose by 1 GB per day at a customer site. I could set up the scenario in our test center, and could get a memory leak of some 700 MB per day.

This application is a Windows service written in C# which communicates with devices over a CAN bus.

The memory leak does not depend on the rate of data the application writes to the CAN bus. But it clearly depends on the number of messages received.

The "unmanaged" side of reading the messages is:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct CAN_MSG
{
    public uint time_stamp;
    public uint id;
    public byte len;
    public byte rtr;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
    public byte[] a_data;
}

[DllImport("IEICAN02.dll", EntryPoint = "#3")]
public static extern int CAN_CountMsgs(ushort card_idx, byte can_no, byte que_type);
//ICAN_API INT32 _stdcall CAN_CountMsgs(UINT16 card_idx, UINT8 can_no,UINT8 que_type);

[DllImport("IEICAN02.dll", EntryPoint = "#10")]
public static extern int CAN_ReadMsg(ushort card_idx, byte can_no, ushort count, [MarshalAs(UnmanagedType.LPArray), Out()] CAN_MSG[] msg);
//ICAN_API INT32 _stdcall CAN_ReadMsg(UINT16 card_idx, UINT8 can_no, UINT16 count, CAN_MSG* p_obj);

We use essentially as follows:

private void ReadMessages()
{
    while (keepRunning)
    {
        // get the number of messages in the queue
        int messagesCounter = ICAN_API.CAN_CountMsgs(_CardIndex, _PortIndex, ICAN_API.CAN_RX_QUE);
        if (messagesCounter > 0)
        {
            // create an array of appropriate size for those messages
            CAN_MSG[] canMessages = new CAN_MSG[messagesCounter];
            // read them
            int actualReadMessages = ICAN_API.CAN_ReadMsg(_CardIndex, _PortIndex, (ushort)messagesCounter, canMessages);
            // transform them into "our" objects
            CanMessage[] messages = TransformMessages(canMessages);
            Thread thread = new Thread(() => RaiseEventWithCanMessages(messages))
            {
                Priority = ThreadPriority.AboveNormal
            };
            thread.Start();
        }
        Thread.Sleep(20);
    }
}

 // transformation process:
new CanMessage
{
    MessageData = (byte[])messages[i].a_data.Clone(),
    MessageId = messages[i].id
};

The loop is executed once per every ~30 milliseconds.

When I call RaiseEventWithCanMessages(messages) in the same thread, the memory leak disappears (well, not completely, some 10 MB per day - ie about 1% of the original leak - remain, but that other leak is likely unrelated).

I do not understand how this creation of threads can lead to a memory leak. Can you provide me with some information how the memory leak is caused?

Addendum 2018-08-16: The application starts of with some 50 MB of memory, and crashes at some 2GB. That means, that Gigabytes of memory are available for most of the time. Also, CPU is at some 20% - 3 out of 4 cores are idle. The number of threads used by the application remains rather constant around ~30 threads. Overall, there are plenty of resources available for the Garbage Collection. Still, GC fails.

With some 30 threads per second, and a memory leak of 700 MB per day, on average ~300 bytes of memory leak per freshly created thread; with ~5 messages per new thread, some ~60bytes per message. The "unmanaged" struct does not make it into the new thread, its contents are copied into a newly instantiated class.

So: why does GC fail despite the enormous amount of resources available for it?

Answer 1

You're creating 2 arrays and a thread every ~30 milliseconds, without any coordination between them. The arrays could be a problem, but frankly I'm much more worried about the thread - creating threads is really, really expensive . You should not be creating them this frequently.

I'm also concerned about what happens if the read loop is out-pacing the thread - ie if RaiseEventWithCanMessages takes more time than the code that does the query/sleep. In that scenario, you'd have a constant growth of threads. And you'd probably also have all the various RaiseEventWithCanMessages fighting with each-other.

The fact that putting RaiseEventWithCanMessages inline "fixes" it suggests that the main problem here is either the sheer number of threads being created (bad), or the many overlapping and growing numbers of concurrent RaiseEventWithCanMessages .

---

The simplest fix would be: don't use the extra threads here.

If you actually want concurrent operations, I would have exactly two threads here - one that does the query, and one that does whatever RaiseEventWithCanMessages is, both in a loop. I would then coordinate between the threads such that the query thread waits for the previous RaiseEventWithCanMessages thing to be complete, such that it hands it over in a coordinated style - so there is always at most one outstanding RaiseEventWithCanMessages , and you stop running queries if it isn't keeping up.

Essentially:

CanMessage[] messages = TransformMessages(canMessages);
HandToConsumerBlockingUntilAvailable(messages); // TODO: implement

with the other thread basically doing:

var nextMessages = BlockUntilAvailableFromProducer(); // TODO: implement

A very basic implementation of this could be just:

void HandToConsumerBlockingUntilAvailable(CanMessage[] messages) {
    lock(_queue) {
        if(_queue.Length != 0) Monitor.Wait(_queue); // block until space
        _queue.Enqueue(messages);
        if(queue.Length == 1) Monitor.PulseAll(_queue); // wake consumer
    }
}
CanMessage[] BlockUntilAvailableFromProducer() {
    lock(_queue) {
        if(_queue.Length == 0) Monitor.Wait(_queue); // block until work
        var next = _queue.Dequeue();
        Monitor.Pulse(_queue); // wake producer
        return _next;
    }
}
private readonly Queue<CanMessage[]> _queue = new Queue<CanMessage[]>;

This implementation enforces that there is no more than 1 outstanding unprocessed Message[] in the queue.

This addresses the issues of creating lots of threads, and the issues of the query loop out-pacing the RaiseEventWithCanMessages code.

I might also look into using the ArrayPool<T>.Shared for leasing oversized arrays (meaning: you need to be careful not to read more data than you've actually written, since you might have asked for an array of 500 but been given one of size 512), rather than constantly allocating arrays.