A SelfHosted WCF Service over Basic HTTP Binding doesn't support more than 1000 concurrent requests

Question

I have self hosted a WCF Service over BasicHttpBinding consumed by an ASMX Client . I'm simulating a concurrent user load of 1200 users . The service method takes a string parameter and returns a string. The data exchanged is less than 10KB. The processing time for a request is fixed at 2 seconds by having a Thread.Sleep(2000) statement. Nothing additional. I have removed all the DB Hits / business logic.

The same piece of code runs fine for 1000 concurrent users. I get the following error when I bump up the number to 1200 users.

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
   --- End of inner exception stack trace ---
   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at WCF.Throttling.Client.Service.Function2(String param)

This exception is often reported on DataContract mismatch and large data exchange. But never when doing a load test. I have browsed enough and have tried most of the options which include,

1. Enabled Trace & Message log on server side. But no errors logged.
2. To overcome Port Exhaustion MaxUserPort is set to 65535, and TcpTimedWaitDelay 30 secs.
3. MaxConcurrent Calls is set to 600, and MaxConcurrentInstances is set to 1200.
4. The Open, Close, Send and Receive Timeouts are set to 10 Minutes.
5. The HTTPWebRequest KeepAlive set to false.

I have not been able to nail down the issue for the past two days.

Any help would be appreciated.

Thank you.

Answer 1

If there are no errors in the service-side WCF logs, I suspect you are hitting some kind of limit in the HTTP.SYS driver layer, leading to requests being turned away before the service application sees them. I think the default limit in the request queue for a particular application may be 1000.

I'm no expert on HTTP.SYS but you may get some insight by running:

netsh http show servicestate

Answer 2

I have seen similar problems on different servers, depending on their CPUs and RAM. You did not mention the server type, how it was upgraded (XP Pro or Server 2003 upgraded to Server 2008), etc. The way I had resolved the issue was through checking the x:\\Windows\\Microsoft.NET\\Framework[version]\\config\\machine.config. Apparently selecting "unlimited" connection through IIS does not mean "unlimited" connections. The number of connections I ran into errored out after 11 requests at the exact same millisecond.

The issue was related to the number of connections coming from the same source. The performance benchmark tool resided on the same PC which has the same IP. The machine.config contains a constraint on the number of connections from the same source.