How to Improve ASP.Net Web Service performance

Question

We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,

1)receives a request with input data from external clients

2)Will get 20 possibilities from database for one request based on input data after validation

3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data. Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.

4)After that Service will send back the all 20 solutions to the client.

This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds) considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.

So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.

I also tried async foreach in place of parallel.foreach as mentioned in below article but still performance is really bad.

https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239

As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.

I think service should not take more than 20 seconds for bulk request if it is lessa than 5 seconds for single request.

Can anyone please me what should be the way forward here to improve the performance ?

Thanks in advance.

Regards,

Raghu.

Answer 1

Looks like a lot of things happening here at the same time. I believe you have on nderlying issue that causes many side effects.

I will make the assumption that your server is sufficient in terms of CPU and memory to handle the concurrent connections (though the CPU 100% makes me wonder).

It seems to me that your problem, is that the parallel tasks (or threads), compete for the same resources. That would explain why multiple requests take much more time and why the async paradigm takes even more.

Let me explain:

The problem in practice

Parallel implementation: 1 or 2 request need minimum synchronization, so even if they compete for the same resources, it should be fine.

When 20 threads, try to access the same resources, a lot is happening and you come to a situation known as livelock.

When you switch to async, no requests await for a thread (they are waiting on the IO threads), so you make the problem even worse.

(I suspect that the problem is on your database. If your database server is the same machine, it would also explain the utilization).

The solution

Instead of trying to up the parallelism, find the contested resources and identify the problem.

If it's in your database (most probable scenario), then you need to identify the queries causing the trouble and fix them (indexes, statistics, query plans and whatnot). DB profilers showing locks and query execution plans are your friends for this.

If the problem is in your code, try to minimize the race conditions and imporve your algorithms.

To get a hint of where to look for, use the Visual Studio profiling tools: https://docs.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019 or any external .net profiling software.