Synergy dbr.exe getting System.ServiceModel.FaultException during xcall

Question

We have Synergy application that uses Synergy's xcall function to load a (COM interop?) assembly (ImageRetrieval.dll) and call methods from it.

This particular method call is a service call to a web service (ImageRetrievalService.asmx) running in DotNetNuke hosted on another server at the same location.

Before we updated the web service, this call was working just fine, but we changed the application pool from the 2.0 CLR to the 4.0 CLR as part of updating our projects to leverage the new (in .net 4.0) Task Parallel Library. We also modified the web.config for the web application, but we didn't remove anything, we only added an assembly name after a type name in a reference, and added the preCondition="runtimeVersionv2.0" to the svc-Integrated handler.

The Synergy application is blowing up with a Watson Bucket problem signature of the following

Description:
  Stopped working

Problem signature:
  Problem Event Name: CLR20r3                
  Problem Signature 01: dbr.exe             //EXE file name
  Problem Signature 02: 9.5.1.3             //EXE file assembly version number
  Problem Signature 03: 4e95fcb0            //EXE File Stamp
  Problem Signature 04: System.ServiceModel //Faulting assembly version
  Problem Signature 05: 4.0.0.0             //Faulting assembly version
  Problem Signature 06: 5073e42b            //Faulting assembly timestamp
  Problem Signature 07: 575d                //Faulting assembly method def
  Problem Signature 08: 205                 //Faulting method IL Offset within the faulting method
  Problem Signature 09: System.ServiceModel.FaultException //Exception type
  OS Version:      6.0.6002.2.2.0.272.7
  Locale ID:           1033

After using ILDASM to decompile the System.ServiceModel 4.0.0.0 assembly and look up the method with signature / 0600575d / I was able to find out it is the System.ServiceModel.Dispatcher.BufferedReceiveBinder.TryReceive method that the exception is occurring in.

public bool TryReceive(TimeSpan timeout, out RequestContext requestContext)
{
    if (Interlocked.CompareExchange(ref this.pendingOperationSemaphore, 1, 0) == 0)
    {
        ActionItem.Schedule(tryReceive, this);
    }

    RequestContextWrapper wrapper;
    bool success = this.inputQueue.Dequeue(timeout, out wrapper);

    if (success && wrapper != null)
    {
        requestContext = wrapper.RequestContext;
    }
    else
    {
        requestContext = null;
    }

    return success;
}

I can't tell whether this is happening in the Synergy application, the ImageRetrieval.dll, or the ImageRetrievalService.asmx, but it sounds like it might be making the jump from ImageRetrieval.dll to ImageRetrievalService.asmx, but for some reason it doesn't understand the message?

The IL decompiled (I don't see an IL offset that matches the offset from Problem Signature 08, 205)

.method /*0600575D*/ private hidebysig static 
        void  TryReceive(object state) cil managed
{
  // Code size       108 (0x6c)
  .maxstack  3
  .locals /*11000D10*/ init (class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/ V_0,
           class System.ServiceModel.Channels.RequestContext/*0200009B*/ V_1,
           bool V_2,
           class [mscorlib/*23000001*/]System.Exception/*0100000A*/ V_3)
  IL_0000:  ldarg.0
  IL_0001:  castclass  System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/
  IL_0006:  stloc.0
  IL_0007:  ldc.i4.0
  IL_0008:  stloc.2
  .try
  {
    .try
    {
      IL_0009:  ldloc.0
      IL_000a:  ldfld      class System.ServiceModel.Dispatcher.IChannelBinder/*020008F4*/ System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/::channelBinder /* 04002C8D */
      IL_000f:  ldsfld     valuetype [mscorlib/*23000001*/]System.TimeSpan/*01000002*/ [mscorlib/*23000001*/]System.TimeSpan/*01000002*/::MaxValue /* 0A000133 */
      IL_0014:  ldloca.s   V_1
      IL_0016:  callvirt   instance bool System.ServiceModel.Dispatcher.IChannelBinder/*020008F4*/::TryReceive(valuetype [mscorlib/*23000001*/]System.TimeSpan/*01000002*/,
                                                                                                               class System.ServiceModel.Channels.RequestContext/*0200009B*/&) /* 060055CD */
      IL_001b:  brfalse.s  IL_0030

      IL_001d:  ldloc.0
      IL_001e:  ldfld      class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/> System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/::inputQueue /* 04002C8E */
      IL_0023:  ldloc.1
      IL_0024:  newobj     instance void System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/::.ctor(class System.ServiceModel.Channels.RequestContext/*0200009B*/) /* 06005762 */
      IL_0029:  ldnull
      IL_002a:  callvirt   instance bool class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/>/*1B00059B*/::EnqueueWithoutDispatch(!0,
                                                                                                                                                                                                                                                                                class [mscorlib/*23000001*/]System.Action/*0100001D*/) /* 0A00203F */
      IL_002f:  stloc.2
      IL_0030:  leave.s    IL_004d

    }  // end .try
    catch [mscorlib/*23000001*/]System.Exception/*0100000A*/ 
    {
      IL_0032:  stloc.3
      IL_0033:  ldloc.3
      IL_0034:  call       bool [System.ServiceModel.Internals/*23000002*/]System.Runtime.Fx/*010000DA*/::IsFatal(class [mscorlib/*23000001*/]System.Exception/*0100000A*/) /* 0A0000CE */
      IL_0039:  brfalse.s  IL_003d

      IL_003b:  rethrow
      IL_003d:  ldloc.0
      IL_003e:  ldfld      class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/> System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/::inputQueue /* 04002C8E */
      IL_0043:  ldloc.3
      IL_0044:  ldnull
      IL_0045:  callvirt   instance bool class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/>/*1B00059B*/::EnqueueWithoutDispatch(class [mscorlib/*23000001*/]System.Exception/*0100000A*/,
                                                                                                                                                                                                                                                                                class [mscorlib/*23000001*/]System.Action/*0100001D*/) /* 0A002040 */
      IL_004a:  stloc.2
      IL_004b:  leave.s    IL_004d

    }  // end handler
    IL_004d:  leave.s    IL_006b

  }  // end .try
  finally
  {
    IL_004f:  ldloc.0
    IL_0050:  ldflda     int32 System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/::pendingOperationSemaphore /* 04002C8F */
    IL_0055:  ldc.i4.0
    IL_0056:  call       int32 [mscorlib/*23000001*/]System.Threading.Interlocked/*01000291*/::Exchange(int32&,
                                                                                                        int32) /* 0A0009AE */
    IL_005b:  pop
    IL_005c:  ldloc.2
    IL_005d:  brfalse.s  IL_006a

    IL_005f:  ldloc.0
    IL_0060:  ldfld      class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/> System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*/::inputQueue /* 04002C8E */
    IL_0065:  callvirt   instance void class [System.ServiceModel.Internals/*23000002*/]System.Runtime.InputQueue`1/*01000020*/<class System.ServiceModel.Dispatcher.BufferedReceiveBinder/*02000912*//RequestContextWrapper/*02000913*/>/*1B00059B*/::Dispatch() /* 0A002041 */
    IL_006a:  endfinally
  }  // end handler
  IL_006b:  ret
} // end of method BufferedReceiveBinder::TryReceive

Edit : Forgot to add the Event Viewer log

Application: dbr.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ServiceModel.FaultException
Stack:

Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
   at ImageRetrievalLib.DataService.DataServiceSoap.RetrieveDocuments(ImageRetrievalLib.DataService.RetrieveDocumentsRequest)
   at ImageRetrievalLib.DataService.DataServiceSoapClient.ImageRetrievalLib.DataService.DataServiceSoap.RetrieveDocuments(ImageRetrievalLib.DataService.RetrieveDocumentsRequest)
   at ImageRetrievalLib.DataService.DataServiceSoapClient.RetrieveDocuments(Byte[], Byte[])
   at ImageRetrievalLib.DocumentRetrieval.RetrieveDocuments(System.String, System.String, System.String, System.String)
   at ImageRetrievalLib.ImageRetrieval.RetrieveDocuments(System.String, System.String, System.String, System.String)

Answer 1

This ended up just being caused by an incorrect Identity property set for the app pool in IIS on the machine that was hosting the ImageRetrievalService.asmx.

The Identity was set to ApplicationPoolIdentity, and that is apparently a temporary profile, so since we changed the app pool CLR from 2.0 to 4.0, ASP.NET 4.0 doesn't allow temporary profiles to validate with the DSACryptoServiceProvider that the service used.