Targetting memory leak in a .NET production service

Question

I have a C#.NET service running in production. The service functions as a TCP server to which clients register and make requests against. In looking at the Task Manager, it appears to be leaking about 10MB/day. I don't seem to notice these in dev (perhaps because of far less traffic and client activity). In searching around I've read that the Task Manager can be seriously wrong, but I'm not sure how accurate this is or in what circumstances the TM would display incorrect information.

To solve this problem I need to more closely monitor memory consumption. The problem is that the leak only seems to appear in production, where the deployed service was built for Release. Also since it's a service that can't be run directly be VS with an attached profiler/debugging, I'm not sure how to best pinpoint the problem with something more precise than TM.

Any group wisdom would be much appreciated, thanks.

EDIT:

• I've added perfmon counters for the privates bytes of the service (7MB to start out) as well as CLR mem in all heaps (30MB to start out)
• Task manager says the total memory to be ~37MB so this seems to make sense
• The first part of this is to let the service go for a day and check out my counters again.

If my private bytes get huge but CLR mem is roughly static this would indicate an unmanaged leak. If both get huge then it's a managed leak.

Thanks guys.

Answer 1

Your first task is figuring out if the process is leaking memory. You can do this with perfmon measuring the Private Bytes http://www.goldstarsoftware.com/papers/CapturingVirtualBytesToALogFile.pdf

If the graph is consistently rising (for say half an hour ) you have a memory leak. You can then use other counters to figure out if this is a .NET leak (.NET memory) though this is unlikely. I find that in most of these cases, there is a COM component that is being invoked but not released.

If you truly have a memory leak (and this isn't just variable memory usage)- the process will shutdown with an out of memory exception after running for a while.

Answer 2

You need one of the below MemoryProfilers in order to monitor it;

http://www.jetbrains.com/profiler/

http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

There are other choices but these are very capable and you can profile remote application's memory with them (at least JetBrains's solution handles that)

Answer 3

Follow this guide: http://blogs.msdn.com/b/tess/archive/2008/03/25/net-debugging-demos-lab-7-memory-leak.aspx

It goes over exactly what you're describing, a memory leak in production. As was mentioned you have to first determine whether it's unmanaged code or managed code that's leaking using perfmon and Private Bytes.

In general make sure for networking objects you're wrapping them in using statements so that they're properly disposed.

A workflow I often use for managed memory leaks is to start the server on a test machine, hit it with a known amount of connections (say 123,456 connections). Then take a memory snapshot by going to task manager and right clicking on the process name and selecting 'create dump'. Open this dump with WinDBG and SOS and run the command !dumpheap -stat. Look for objects that have a multiple of 123,456 instances. Should these objects still be in memory? If not run a !gcroot on an instance of those objects to find why it's still in memory.

Answer 4

1. Get a dump of the memory when its in a leak state using the Task Manager right click on the process and select create dump file. You can also use ProcDump which gives you more options.

2. Use SOS Extensions in either WinDebug or Visual Studio to inspect the memory.