Azure Cloud Service is dead after Application Pool Recycle

Question

I have a deployed Azure Cloud Service WebRole WebAPI with only one instance. I have noticed that if I wait some idle time (No HTTP Requests), Then later on the service is dead, and every request to it results in the following response:

{
    Message: "An error has occurred."
}

The Azure management dashboard says the instance status is Running. Remote desktop reveals that the Application pool and the web site are both running, and that all the services needed are running.
I can't find any relevant error message anywhere...

The only resolution I found to the above situation is to perform a reboot to the Instance VM.

Since I assumed that the above is a result of idle time, I looked and found out that by default the Application Pool itself has an Idle Timeout property set to 20 minutes, resulting in shutting down the worker process inside the Application Pool after idle timeout reached. I managed to reproduce the problem without waiting for the timeout by manually triggering Application Pool Recycling.

I assume that I have some kind of problem with my service, unable to correctly recover from shut down. As a proof, A clean out-of-the-box service project did not result with the same behavior.

What can cause such a problem?
Can anyone point me to the correct direction of debugging this?
Where can I possibly find a helpful error trace message?

Edit :
I am using Autofac for IoC, and it suddenly occurred to me that the problem might be that the IoC is somehow cleared after the recycle. I am performing the IoC registrations inside Application_Start() of Global.asax.cs .
Could it be the problem?

Answer 1

I finally found the problem source, and it is indeed related to the IoC container.

In order to reproduce the issue locally, I deployed the Cloud Service to a local IIS server (instead of IIS Express). I can now Stop/Start the WebSite, and actually see the exception stack trace, which right away proved that the issue is actually realted to unregistered assemblies in the IoC.

Apparently, whenever the application pool recycles, all the loaded assemblies are cleared and not reloaded according to the same rules. I actually found an assembly (that is a weak-referenced project in my solution) that was not loaded at the moment IoC was re-initializing.

I am not sure why it behaves differently from local deployment, but I managed to fix my problem by calling:

Assembly.Load([Assembly Name]);

before initializing the IoC.

Answer 2

When the AppDomain is recycled by IIS assemblies are only loaded on demand. Try using the GetReferencedAssemblies method on System.Web.Compilation.BuildManager to get a list of the referenced assemblies instead.

var assemblies = BuildManager.GetReferencedAssemblies().Cast<Assembly>();

That should force the referenced assemblies to be loaded into the AppDomain immediately making them available for module scanning.