WCF hosted in IIS has lots of open requests and the service is slow down

Question

I have a WCF web service based on JSON and POST method which have a function called website. This function has a simple code and only call another web service using the code bellow:

        using (var cli = new MyWebClient())
        {                
            Task<string> t = cli.UploadStringTaskAsync(myURI, "POST", request);
            if (t == await Task.WhenAny(t, Task.Delay(400)))
            {
                response = t.Result;
            }
            else
            {
                response = "";
            }
            cli.Dispose();
        }

and MyWebClient class is implemented as:

class MyWebClient : WebClient
        {
            protected override WebRequest GetWebRequest(Uri address)
            {
                WebRequest request = base.GetWebRequest(address);
                if (request is HttpWebRequest)
                {
                    (request as HttpWebRequest).KeepAlive = true;                    
                    (request as HttpWebRequest).ContentType = "application/json";
                }
                return request;
            }
        }

The problem is that I can see in IIS that a large number of requests remain open more than 18 seconds and even more for 1 or 2 of my worker processes(as you can see in the attached image for one of them)!!! And this makes the service very slow down. Note that this service has about 2K requests per second and the application pool of this service has a web garden containing 12 worker processes and the queue limit is 10K. This situation takes place when there are (for example) 4 worker processes working in a predictable time(about 450 ms) and IIS shows that the maximum elapsed time on their requests are about 380.

a large number of requests that remain open in the IIS

Note that I have used cli.UploadStringTaskAsync and hence timeout is not considered for cli . So, I have to implement a code like t == await Task.WhenAny(t, Task.Delay(400)) to simulate timeout.

What is the problem do you think?! Does using await cause many context switch and the requests are queued to be executed by cpu?

---

Edit:

Here you can find some recommendation that is helpful, but no one can help and solve the problem. I set them up in my application's web config but it couldn't resolve my issue.

---

Update:

As additional information note that the network card is 1G, and at most we have 100Mgb/s bandwidth usage. There are 6 cores and 12 logical processors of Intel Xeon E5-1650 V3 3.5 Ghz. We have 128GB of RAM and 480GB of SSD.

Answer 1

I found a solution that solves the problem. The key point was "processModel Element (ASP.NET Settings Schema)". As I mentioned in my question:

This situation takes place when there are (for example) 4 worker processes working in a predictable time(about 450 ms) and IIS shows that the maximum elapsed time on their requests are about 380.

So, I think balancing the load among worker processes could be the problem. By configuring the processModel Element manually I have solved the issue. After researching a lot I found this valuable link about processModel Element and its properties.

Also this link describes all the properties and the effect of each item. As mentioned in this link there are 2 important properties called "requestLimit" and "requestQueueLimit":

requestQueueLimit : Specifies the number of requests that are allowed in the queue before ASP.NET begins returning the message "503 – Server Too Busy" to new requests. The default is 5000.

requestLimit : Specifies the number of requests that are allowed before ASP.NET automatically launches a new worker process to take the place of the current one. The default is Infinite.

The solution is to control and limit requestLimit by a rational number for example 300 in my case. Also, limiting requestQueueLimit by multiplication of the number of worker processes and requestLimit . I have increased the number of worker process to 20 and by this configuration 6000 request can be queued totally and each worker process has 300 request at most. By reaching 300 request per worker process ASP.NET automatically launches a new worker process to take the place of the current one.

So the load is balanced better among the worker processes. I have checked all queues and there is no request with more than 400 time elapsed!!!

I think this solution could be used as a semi-load balancer algorithm for IIS and worker processes by playing with these properties( requestLimit , requestQueueLimit and number of worker processes ).