Debugging a VB6 COM+ Web Application Hang

Question

I work on a Classic ASP web application that uses several old COM components written in VB6. All of the VB6 components are registered in a COM+ application that run in their own dllhost process. A large majority of the application has been converted to .Net, but there are still a lot of legacy pages and components. COM Interop is used in both directions, calling some .Net assemblies from classic ASP and VB6 as well as calling VB6 components from ASP.Net. The application is running on Windows Server 2008 R2 (IIS 7.5) in the classic pipeline mode.

For the most part the application works fine. The transition to .Net effort was ultimately abandoned, with a new product being developed instead. In the meantime, the old product must be maintained in it's heterogeneous state.

I am having trouble tracking down an intermittent problem where the web application hangs. Users just see a blank screen while their browser waits and the server never responds. The hang persists until I manually kill the dllhost process that's hosting the VB6 components, so I believe the problem is buried there. Probably a memory leak or runaway circular loop.

There are thousands of users on the system daily, but the problem only happens once or twice a week. Fortunately we have a web farm that automatically pulls a server out when it stops responding, so the customer impact is zero. Still, I would like to figure out what's going on.

I have recompiled all of the VB6 components to include debugging symbols and redeployed to production. When the problem happens, I use the 32-bit task manager (c:\\windows\\syswow64\\taskmgr.exe) to take a crash dump of the dllhost process. I end up with a dllhost.dmp file, which I bring down to my development workstation and open in VS2010. I have the .pdb symbol files that VB6 created in my symbols path. When I start the debugging session in VS2010, I can go to the Modules screen and see that indeed all of the symbols for my components are loaded.

Where to go from here? The call stack doesn't show any of my own components. It looks like this:

The disassembly at the top of the call stack looks like this:

Not sure what else I can do. I examined all of the locals at every frame of the call stack and it's gibberish to me. I don't see any references to any of my own components.

Perhaps WinDbg would yield more information? Not sure where to start with it.

I'm pretty sure that if I could just find what VB6 class/method was being called when the hang occurred that I could get to the bottom of it. I've tried adding some logging, and the results are inconsistent.

Perhaps there's nothing wrong with my VB6 components at all, but I'm hitting on some bug within Windows or IIS?

Any advice would be appreciated, but throwing away VB6 is not an option at this point. Thanks.

Answer 1

Not a complete answer, but CoRegisterSurrogateEx is documented to block as long as the surrogate process is running:

The CoRegisterSurrogateEx function is a blocking function. It does not return until COM+ has determined that the process will be shut down. Before calling this function, initialize COM on this thread as a multi-threaded apartment (MTA).

So I don't think the error is on this call-stack. (You can see it was still blocking on a WaitForSingleObject call, most likely the mechanism it uses to block until the process is to be shut down).