Pinned memory causes fragmentation in C# WebApi

Question

We have a web application that works as a reverse proxy (forwarding requests and responses). This is part of a bigger platform so quite a large number of calls are getting through there. The application is run on IIS and we use ASP.NET WebApis ApiController to receive and HttpClient to resend the calls (newer NuGet packages on .NET 4.6.1). We have disabled IIS worker process recycling.

In production we can see that over about one week the memory usage is constantly growing and then remaining at a high level. Occasionally we are getting OutOfMemoryExceptions. Using R# dotMemory profiler we couldn't find a memory leak but we can see that there is a huge fragmentation (up to 99%!) of Gen 0 to 2 objects. We assume this is related to the OverlappedData objects that dotMemory marks as pinned.

There are some posts out there dealing with this kind of issue, a good one being http://www.ahuwanya.net/blog/post/Buffer-Pooling-for-NET-Socket-Operations - the summary being that the fragmentation is related to the buffer pinning for the IO operations. So on this ground we have some questions:

1. How could we introduce a buffer pool as shown in the post with Sockets for use with ApiController and HttpClient?
2. How could we force pinned memory to be released, for example by closing a connection - without having a significant performance loss?
3. How can we ensure the assumption is correct by getting more detailed information about the pinned memory? We found something about logging stacktraces with Perfmon but a more detailed guide would be appreciated.

Of course any other suggestions, hints or remarks that help us to narrow down and solve the issue are appreciated as well. Please note that we want to solve the issue and not have a workaround for it (such as restarting the IIS worker process every day).

Answer 1

In our current release we don't have this problem anymore. We have no real explanation for it - the only thing that seems to have changed is that we removed some "volatile" definitions. Could this have been the cause? Maybe yes - in fact we assume it is, but have no real proof for it. According to the documentation it is somewhat unclear whether volatile will prevent memory optimization, but from understanding it is likely. So we just accept it like this...