IIS App Pool crashing on recycle

Question

I hope someone can help with this very strange situation.

The w3wp process (for any site on our server) crashes with an Access Violation exception when it's recycled. This happens whether it recycles due to a time or request limit, or a manual trigger. It doesn't seem to be related to our application, as a dummy site with no content exhibits the same behaviour.

There are 2 servers in production with this behaviour, using NLB and ARR to load balance to an IIS server farm. We have 2 test servers with the same setup on our local infrastructure which also have this problem, but "single server" setups (and developer machines) of the same application don't have this problem.

The Event Logs of these errors are like the following:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000001
Faulting process id: 0x32d8
Faulting application start time: 0x01d34d9dc16ec0b0
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: unknown
Report Id: f3b2821a-b991-11e7-9902-00505683efbb

Sometimes they have a different Faulting module, but mostly it's "unknown":

Faulting module name: nativerd.dll, version: 7.5.7601.17855, time stamp: 0x4fc85321

or

Faulting module name: iiscore.dll, version: 7.5.7601.17514, time stamp: 0x4ce7c6c8

I have captured crash dumps when these errors occur using DebugDiag 2.2 and the analysis shows the following for the thread where the exception was raised:

Thread 0 - System ID 16208

    Entry point   w3wp!wmainCRTStartup 
    Create time   25/10/2017 09:19:17 
    Time spent in user mode   0 Days 00:00:00.078 
    Time spent in kernel mode   0 Days 00:00:00.015 

        Call Stack

        iiscore!W3_URL_INFO::`vftable' 
        nativerd!TerminateNativeConfiguration+16 
        w3wphost!W3WP_HOST::~W3WP_HOST+1fb 
        w3wphost!AppHostInitialize+325 
        w3wp!wmain+470 
        w3wp!PerfStopProvider+19b 
        kernel32!BaseThreadInitThunk+d 
        ntdll!RtlUserThreadStart+1d 

I've also looked at this in WinDbg and got the following:

OS Thread Id: 0x3f50 (0)
Current frame: iiscore!W3_URL_INFO::`vftable'
Child-SP         RetAddr          Caller, Callee
00000000001df570 000007fef91ef4d6 nativerd!TerminateNativeConfiguration+0x16
00000000001df5a0 000007fefb1c4797 w3wphost!W3WP_HOST::~W3WP_HOST+0x1fb, calling 0000000000010000
00000000001df5d0 000007fefb1c4269 w3wphost!AppHostInitialize+0x325, calling w3wphost!W3WP_HOST::~W3WP_HOST
00000000001df630 00000000ffbe3c60 w3wp!wmain+0x470
00000000001df670 000007feff3414e4 msvcrt!calloc_impl+0x85, calling ntdll!RtlAllocateHeap
00000000001df720 000007feff3541ba msvcrt!_wgetmainargs+0x7b, calling msvcrt!wsetenvp
00000000001df750 00000000ffbe10a2 w3wp!PerfStopProvider+0x4c, calling msvcrt!_wgetmainargs
00000000001df770 000000007715df6a ntdll! ?? ::FNODOBFM::`string'+0x149ca, calling ntdll!NtQueryPerformanceCounter
00000000001df790 000007feff348e47 msvcrt!initterm+0x1f
00000000001df7c0 00000000ffbe11f1 w3wp!PerfStopProvider+0x19b, calling w3wp!wmain
00000000001df7d0 00000000ffbe1351 w3wp!wmainCRTStartup+0x9, calling w3wp!_security_init_cookie
00000000001df800 0000000076ec59cd kernel32!BaseThreadInitThunk+0xd
00000000001df830 00000000770fa561 ntdll!RtlUserThreadStart+0x1d

But I'm struggling with how to interpret this.

Any insight into what this might be would be, or how I could continue the diagnosis would be much appreciated.

Many thanks

Answer 1

On the off-chance that this helps someone else, we finally tracked this down to an extra IIS module, Dionach StripHeaders that was causing the access violation exception. It was a known bug which has been fixed, so updating the module should fix our issue.