Random TimeoutException in client on many concurrent Async Calls

Question

I have a TimeoutException problem, I am using C# 4.0 (can't upgrade to 4.5 anytime soon) and WCF. Note that I do not control the Server and cannot see the code and or technology that are used. The problem happens with different servers made by different people.

I send as many request as I can to many servers (let's say 10), one per server at any time. They go from 2 to 30 requests per second. Between 30 seconds to 5 minutes, I will get some TimeoutException :

exception   {"The HTTP request to 'http://xx.xx.xx.xx/service/test_service' has exceeded the allotted timeout of 00:02:10. The time allotted to this operation may have been a portion of a longer timeout."}   System.Exception {System.TimeoutException}.

Stack Trace :
Server stack trace: 
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult     result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg,  IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Device.EndTest(IAsyncResult result)
   at DeviceClient.EndTest(IAsyncResult result) in ...
   at TestAsync(IAsyncResult ar) in ...

The InnerException is :

[System.Net.WebException]   {"The request was aborted: The request was canceled."}  System.Net.WebException
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)

Wireshark tells me that I do not even open a connection (no SYN). So this should be a client problem. I have many TIME_WAIT connections in TCPView

Using Sync calls is working, but not possible.

Note that in the following code example, there is one method call per server. (In my case, 10 concurrent TestAsync)

(in the real project, we use CCR instead of Semaphore, same result)

private void AsyncTest()
{   
    //GetServiceObject Will add custom bindings and more..
    Client client = ClientBuilder.GetServiceObject();

    while (true)
    {
         Semaphore semaphore = new Semaphore(0,1);
         client.BeginTest(BeginTestCallback, new AsyncState
         {
             Client = client,
             Semaphore = semaphore
         });

         semaphore.WaitOne();   
    }   
 }

private void BeginTestCallback(IAsyncResult asyncResult)
{
    try
    {
        AsyncState state = asyncResult.AsyncState as AsyncState;

        Client client = state.Client;
        Semaphore semaphore = state.Semaphore;

        Client.EndTest(asyncResult);    

        semaphore.Release();
    }
    catch (Exception e)
    {
        //Will catch the exception here because of Client.EndTest(asyncResult)
        Debug.Assert(false, e.Message);
    }
}

I tried with

ServicePointManager.DefaultConnectionLimit = 200;
ServicePointManager.MaxServicePointIdleTime = 2000;

As some post suggested, without success.

Even if I set really High Open, send, receive and close timeouts, it will do the same exception. WCF seems to be "stuck" at sending the request. The server continues to respond correctly to other requests.

Have any idea?

Also, If I do this (BeginTest in Callback instead of while(true)), it will never do the exception?!?!

private void AsyncTest()
{
    //GetServiceObject Will add custom bindings and more..
    Client client = ClientBuilder.GetServiceObject();
    try
    {
        client.BeginTest(BeginTestCallback, new AsyncState
        {
            Client  = client            
        });
    }
    catch (Exception e)
    {
        Debug.Assert(false, e.Message);
    }
}

private void BeginTestCallback(IAsyncResult asyncResult)
{
    try
    {
        AsyncState state = asyncResult.AsyncState as AsyncState;        

        state.Client.EndTest(asyncResult);

        state.Client.BeginTest(BeginTestCallback, state);
    }
    catch (Exception e)
    {
        //No Exception here
        Debug.Assert(false, e.Message);
    }
}

Answer 1

After more testing, I found out that if the begin/end mechanism is not executed on the same thread pool, it will randomly do this behavior.

In the first case, "AsyncTest" was spawned within a new thread with ThreadStart and Thread. In the second case, only the first "begin" is called on the dedicated thread and since the problem occurs at random, there is a small chance that the exception would happen on first request. The other "begin" are made on the .net ThreadPool.

By using Task.Factory.StartNew(() => AsyncTest()) in the first case, the problem is gone.

In my real project, I still use CCR (and the CCR threadpool) to do everything until I have to call the begin/end.. I will use the .net threadpool and everything is working now.

Anyone have better explanation of why WCF doesn't like to be called on another threadpool?