After Deploying ASP.NET 4 web application to IIS6 server, worker process recycles cause it to stop working most of the time

Question

This is a really odd situation, so hopefully I can explain it well enough.

I am deploying an ASP.NET 4 webforms application to a Windows Server 2003 SP2 server running IIS6.

Here's the problem -- when the application pool recycles its worker process (w3wp.exe), about 80% of the time, I will get an ReflectionTypeLoadException error trying to access any page in the app that contains an EntityDataSoure every time I try to view it.

However (this is the interesting part) -- the other 20%, it works just fine. I've actually resorted to turning off recycling the worker process entirely for this application pool and just add/remove whitespace from web.config forcing the site to recompile until I get a "good" w3wp.exe.

If this isn't clear, what I'm saying is: the actual worker process doesn't work at all for pages containing an EntityDataSource for about 4/5 times it starts, but still manages to serve all other pages just fine. Once you get a worker process that manages to serve a page with an EntityDataSource, it works every time until that process gets recycled.

My question is, how can I debug this? It works fine on my dev machine, it works fine on the server as long as you get a good process running, but iisreset or a server restart or anything that kills the worker process is almost guaranteed to cause the site to not come back up and throw this ReflectionTypeLoadException.

Here is the compilation section of my web.config:

<compilation debug="false" targetFramework="4.0">
  <assemblies>
    <add assembly="System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
    <add assembly="System.Design, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
    <add assembly="System.Web.Extensions.Design, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <add assembly="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
  </assemblies>
</compilation>

Here is all the information I have about this exception:

Application information: 
    Application domain: /LM/W3SVC/1/Root/name-5-129230865053805490 
    Trust level: Full 

Process information: 
    Process ID: 3300 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\NETWORK SERVICE 

Exception information: 
    Exception type: ReflectionTypeLoadException 
    Exception message: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

Thread information: 
    Thread ID: 6 
    Thread account name: NT AUTHORITY\NETWORK SERVICE 
    Is impersonating: False 
    Stack trace:    at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
   at System.Data.Metadata.Edm.ObjectItemConventionAssemblyLoader.LoadTypesFromAssembly()
   at System.Data.Metadata.Edm.ObjectItemAssemblyLoader.Load()
   at System.Data.Metadata.Edm.AssemblyCache.LoadAssembly(Assembly assembly, Boolean loadReferencedAssemblies, ObjectItemLoadingSessionData loadingData)
   at System.Data.Metadata.Edm.AssemblyCache.LoadAssembly(Assembly assembly, Boolean loadReferencedAssemblies, KnownAssembliesSet knownAssemblies, EdmItemCollection edmItemCollection, Action`1 logLoadMessage, Object& loaderCookie, Dictionary`2& typesInLoading, List`1& errors)
   at System.Data.Metadata.Edm.ObjectItemCollection.LoadAssemblyFromCache(ObjectItemCollection objectItemCollection, Assembly assembly, Boolean loadReferencedAssemblies, EdmItemCollection edmItemCollection, Action`1 logLoadMessage)
   at System.Data.Metadata.Edm.ObjectItemCollection.ExplicitLoadFromAssembly(Assembly assembly, EdmItemCollection edmItemCollection, Action`1 logLoadMessage)
   at System.Data.Metadata.Edm.MetadataWorkspace.ExplicitLoadFromAssembly(Assembly assembly, ObjectItemCollection collection, Action`1 logLoadMessage)
   at System.Web.UI.WebControls.EntityDataSourceView.ConstructContext()
   at System.Web.UI.WebControls.EntityDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments)
   at System.Web.UI.WebControls.ListControl.OnDataBinding(EventArgs e)
   at System.Web.UI.WebControls.ListControl.PerformSelect()
   at System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
   at System.Web.UI.WebControls.ListControl.OnPreRender(EventArgs e)
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

No changes are made to any files in the project, so I can't figure out why sometimes I get a "good" w3wp.exe and other times I get a "bad" w3wp.exe.

Answer 1

Check that you haven't got another ASP.NET site built against a lower version ( < 4.0 ) running in the same application pool. As with ASP.NET 1.1 and 2.0, ASP.NET 4.0 can't co-reside in the same worker process with other versions.

Update

I didn't catch the part of the exception that reports this because it was out of view:

Exception message: Unable to load one or more of the requested types.
Retrieve the LoaderExceptions property for more information.

^ ^ ^ look up, the framework is helping us for a change ^ ^ ^

Can you wrap a try/catch block around this in your code and dumpy the exceptions you find?

Something like the following code would do:

try
{
  //Do your entity data thing in the page...
}
catch(ReflectionTypeLoadException rtle)
{
  foreach(Exception ex in rtle.LoaderExceptions)
  {
    Debug.WriteLine(ex.ToString());
  }
}

LoaderExceptions is a member of the ReflectionTypeLoadException exception being thrown and contains array of the exceptions being thrown by the class loader.

This should help you narrow things down, if not then post the list of exceptions you find in there.

Answer 2

Here's an update on this horrible bug -- I was deploying a reportviewer control and referencing a couple of the dlls required in my project with copy local = true. It seems that you have to install the ReportViewer redist on the server or you get funny behavior like this problem, which apparently had nothing to do with a EntityDataSource after all.

In short, I fixed this by installing the ReportViewer 10 redist on the server instead of just copying the dlls into bin.

http://www.microsoft.com/downloads/details.aspx?FamilyID=a941c6b2-64dd-4d03-9ca7-4017a0d164fd&displaylang=en

Answer 3

We had a similiar problem and solved it by making sure all projects which were referenced by the project giving the error had referenced the same versions of the dlls. ie project A was throwing this error, and project A referenced dll version 2. Project A also referenced project B, which itself referenced dll version 1. version 2 of the dll was being overwritten by version 1 from project B. As soon as the reference was changed in project b, project a worked like it should.