IIS ASP.net application suddenly hangs requests

Question

We have an asp.net application running on IIS 8.5, when there is high traffic, intermittently by about 1 minute, "requests queue" starts to grow and processor time goes down, look at the performance monitor graph:

The strange thing is that there is no application restart, 503 or IIS recycle occurring, so what could it be? What suddenly makes the IIS hang the requests for a while? Besides the graph, you can see that Windows's memory looks ok and stable.

Here is some environment configurations: APP Pool Queue length: 10000 (there is no 503, so I don't think could it be this)

Asp.net config:

 <system.web>
     <applicationPool maxConcurrentRequestsPerCPU="999999" />
 </system.web>

machine.config:

<processModel autoConfig="false" requestQueueLimit="250000"/>

We configured things this way because our application uses a lot of SignalR.

The application uses Azure SQL Server and Azure Redis, but this is not the problem since another Virtual Machine (with the same APP) doesn't show the problem at the same moment.

Another tip is that: In the same VM, we have the same APP but in another Application Pool and another domain that behaves equal.

Any help would be appreciated, this is driving me crazy.

Thanks!

Answer 1

Did you follow the recommendations regarding configuring the threadpool growth settings that Redis provides, see here . Had similar issues.

Answer 2

In the environment you have described, I will look at below points

1. have a look at DTU percentage for Azure SQL server, if your signalr operations has anything to do with database. just try going one level up in DTU Scale to handle burst. DTU
2. for an app with signalr with multiple virtual machine (assuming load balanced), do you have central signalr server or each server behaving as signalr server ?

if you can install NewRelic or similar, it will point to source of problem.

Hope it helps.

Answer 3

You could take a memory dump of the process when the problem is happening (one way is using Task Manager), although if there are multiple IIS app pools running on the machine, finding the correct process to take a memory dump of might be non-trivial.

If you have Visual Studio Enterprise, then it will give you a nice UI to analyse the dump. It will show you all the threads running the process and what the callstack was at the time of the memory dump. You'll probably find most of the .NET threads have a similar callstack, which would be the most likely cause of the bottleneck.

If you don't have Visual Studio Enterprise, then I think you can still open it with WinDbg, but it's a CLI tool and I don't know the commands, so you'll need to look it up.

Answer 4

Add another counter to your graph: Private Bytes . This number will tell you the number of bytes in all in all managed heaps + the number of unmanaged bytes. If that number keeps increasing, then you have a memory leak somewhere. Ensure all disposable objects are disposed of properly and items are not surviving until G3 collection. Anyhow, I would start there and if this is the issue, I would start my investigation and see what is leaking.