WCF service timing out or shutting down?

Question

I recently noticed in our production environment that exceptions are being logged that WCF services are timing out between calls from my frontend server to ib logic server. Both using WCF to communicate. The frontend server is logging the following exception:

This happens intermittently. What's the cause for this? Is it network related?

Timestamp: 23/12/2013 07:59:00.507895
Message: HandlingInstanceID: 58495715-f5d8-49a9-9c90-763a1718228b
An exception of type 'System.ServiceModel.CommunicationException' occurred and was caught.
------------------------------------------------------------------------------------------
12/23/2013 07:59:00
Type : System.ServiceModel.CommunicationException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message : An error occurred while receiving the HTTP response to https://myservername:2083/IBServer/PayeesService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
Source : Facade
Help link : 
Data : System.Collections.ListDictionaryInternal
TargetSite : System.Collections.Generic.List`1[Facade.BusinessObjects.Payee] GetPayeeList()
Stack Trace : 
Server stack trace: 
   at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at IB.Facade.PayeeFacade.GetPayeeList() in Facade.Payees.cs:line 236
   at LoginBase.RetrievePayees()
   at LoginBase.LoadPayees()
   at Login.LoadDefaultCache()

FRONTEND WCF BINDING to WCF Service on Transaction Server:

 <binding name="WSHttpBinding_ManagePayeeService" closeTimeout="00:02:00" openTimeout="00:02:00" receiveTimeout="00:10:00" sendTimeout="00:02:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">

            <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />

            <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />

            <security mode="Transport">

                <transport clientCredentialType="None" proxyCredentialType="None" realm="" />

                <message clientCredentialType="UserName" algorithmSuite="Default" />

            </security>

        </binding>

WCF BINding on Transaction Server:

 <binding name="WSHttpBinding_ManagePayeeService" closeTimeout="00:02:00" openTimeout="00:02:00" receiveTimeout="00:10:00" sendTimeout="00:02:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
      <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
      <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
      <security mode="Transport">
        <transport clientCredentialType="None" proxyCredentialType="None" realm="" />
        <message clientCredentialType="Windows" negotiateServiceCredential="true" establishSecurityContext="true" />
      </security>
    </binding>

Answer 1

Have you tried looking into the svclogs for getting more information as to what's going wrong?

This logging can be configured in your Web.config under the system.diagnostics element. The below is taken from one of my test files that I occasionally use.

Note: Do not leave this logging on all the time on a production site, as the logs will quickly grow to a problematic size. For this reason, whenever I put the section below into play I comment it out as soon as possible.

When enabling this kind of logging, it works best if you capture both the client and the server logs. Often the server log will have information that isn't passed onto the client, and vice versa.

<system.diagnostics>
  <!--
    Trace Levels:
      Off, Critical, Error, Warning, Information, Verbose, ActivityTracing, All
  -->
  <!--<trace autoflush="true" />-->
  <sources>
    <source name="System.ServiceModel" switchValue="All" propagateActivity="true">
      <listeners>
        <add name="System.ServiceModel" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\System.ServiceModel.svclog" />
      </listeners>
    </source>
    <source name="System.ServiceModel.MessageLogging" switchValue="All">
      <listeners>
        <add name="System.ServiceModel.MessageLogging" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\System.ServiceModel.MessageLogging.svclog" />
      </listeners>
    </source>
    <source name="System.ServiceModel.IdentityModel" switchValue="All">
      <listeners>
        <add name="System.ServiceModel.IdentityModel" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\System.ServiceModel.IdentityModel.svclog" />
      </listeners>
    </source>
    <source name="System.ServiceModel.Activation" switchValue="All">
      <listeners>
        <add name="System.ServiceModel.Activation" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\System.ServiceModel.Activation.svclog" />
      </listeners>
    </source>
    <source name="System.ServiceModel.Serialization" switchValue="All">
      <listeners>
        <add name="System.ServiceModel.Serialization" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\System.ServiceModel.Serialization.svclog" />
      </listeners>
    </source>
    <source name="System.IO.Log" switchValue="All">
      <listeners>
        <add name="System.IO.Log" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\System.IO.Log.svclog" />
      </listeners>
    </source>
    <source name="Cardspace" switchValue="All">
      <listeners>
        <add name="Cardspace" type="System.Diagnostics.XmlWriterTraceListener" initializeData ="C:\Logs\MyWebsiteLogs\SVC\WCF\Cardspace.svclog" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>

You can view the logs using the Service Trace Viewer Tool . This tool may or may not already be installed alongside your current version of Visual Studio / .NET Framework.

EDIT: For context, I find that whenever I get a communication exception like this in WCF it always turns out to be a serialization error of some kind on the server. That's just in my experience however, not sure of what other scenarios could generate an error like this.

Answer 2

I generally get this when a exception is not being handled. Is anything showing up in the event logs? Are you using IIS or self hosting?

Answer 3

看一下MaxRequestLength和readerQuotas，看起来有时默认设置还不够