ASP.NET application hangs under modest load in IIS 7. Requests timeout. How to debug?

Question

We have an ASP.NET site running on IIS7 on Windows Server 2008--actually we run the same code on several identical servers. Traffic has been growing and occasionally, at higher-traffic times, the site (actually the application) will become unresponsive--or at least so slow as to appear unresponsive. The only thing to do is to stop/start W3SVC; recycling the site or application don't fix it.

The same symptoms happen (occasionally and seemingly randomly) on more than one server. So it is not a particular box. These servers have 8GB of RAM, and don't use more than 5GB; and CPU usage averages around 40-50% even in these high-traffic times. So those aren't likely the issue.

There are no application errors or underlying timeouts (SQL, etc.), and nothing is recorded in the Event Log. Normally, when we watch ASP.NET in PerfMon, Requests Executing hovers around 5-10 and Requests Current around 25-40. When this starts, those numbers both steadily climb towards infinity as every request times out for the client.

While I'd love a silver bullet, I'm mainly hoping for a better approach to debugging things. Are there better PerfMon numbers to look at? Something that could be logged? Something to try or test?

Thanks in advance for your ideas, hive-mind!

Answer 1

Tess Ferrandez has a great series of posts/labs on how to debug these kinds of problems:

.NET Debugging Demos - Information and setup instructions

Unfortunately there's rarely a single silver bullet solution to these kinds of problems. You need to get a dump of that worker process when it hangs and then start poking about using WinDBG and SOS.