HttpClient and long running Windows Services - What is the potential for thread deadlock and how to circumvent it

Question

Note: I am running on .NET Framework 4.6.2

Background

I have a long running Windows Services that, once a minute, queues up a series of business related tasks that are ran on their own threads that are each awaited on by the main thread. There can only be one set of business related tasks running at the same time, as to disallow for race conditions. At certain points, each business task makes a series of asynchronous calls, in parallel, off to an external API via an HttpClient in a singleton wrapper. This results in anywhere between 20-100 API calls per second being made via HttpClient.

The issue

About twice a week for the past month, a deadlock issue (I believe) has been cropping up. Whenever it does happen, I have been restarting the Windows Service frequently as we can't afford to have the service going down for more than 20 minutes at a time without it causing serious business impact. From what I can see, any one of the business tasks will try sending a set of API calls and further API calls made using the HttpClient will fail to ever return, resulting in the task running up against a fairly generous timeout on the cancellation token that is created for each business task. I can see that the requests are reaching the await HttpClientInstance.SendAsync(request, cts.Token).ConfigureAwait(false) line, but do not advance past it.

For a additional clarification here, once the first business task begins deadlocking with HttpClient, any new threads attempting to send API requests using the HttpClient end up timing out. New business threads are being queued up, but they cannot utilize the instance of HttpClient at all.

Is this a deadlocking situation? If so, how do I avoid it?

Relevant Code

HttpClientWrapper

public static class HttpClientWrapper
{

  private static HttpClientHandler _httpClientHandler;
  //legacy class that is extension of DelegatingHandler. I don't believe we are using any part of
  //it outside of the inner handler. This could probably be cleaned up a little more to be fair
  private static TimeoutHandler _timeoutHandler;
  private static readonly Lazy<HttpClient> _httpClient =
                                       new Lazy<HttpClient>(() => new HttpClient(_timeoutHandler));
  public static HttpClient HttpClientInstance => _httpClient.Value;

  public static async Task<Response> CallAPI(string url, HttpMethod httpMethod, CancellationTokenSource cts, string requestObj = "") 
  {
    //class that contains fields for logging purposes
    var response = new Response();
    string accessToken;
    var content = new StringContent(requestObj, Encoding.UTF8, "application/json");
    var request = new HttpRequestMessage(httpMethod, new Uri(url));
    if (!string.IsNullOrWhiteSpace(requestObj))
    {
      request.Content = content;
    }

    HttpResponseMessage resp = null;
    
    try
    {
      resp = await HttpClientInstance.SendAsync(request, cts.Token).ConfigureAwait(false);
    }
    catch (Exception ex)
    {
      if ((ex.InnerException is OperationCanceledException || ex.InnerException is TaskCanceledException) && !cts.IsCancellationRequested)
        throw new TimeoutException();
      throw;
    }
    response.ReturnedJson = await resp.Content.ReadAsStringAsync();
    // non-relevant post-call variables being set for logging...
    return response;
  }

  //called on start up of the Windows Service
  public static void SetProxyUse(bool useProxy)
  {
    if (useProxy || !ServerEnv.IsOnServer)
    {
      _httpClientHandler = new HttpClientHandler
      {
        UseProxy = true,
        Proxy = new WebProxy {Address = /* in-house proxy */},
        AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
      };
    }
    else
    {
      _httpClientHandler = new HttpClientHandler
      {
        UseProxy = false,
        AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
      };
    }
    
    _handler = new TimeoutHandler
    {
      DefaultTimeout = TimeSpan.FromSeconds(120),
      InnerHandler = _httpClientHandler
    };
  }
}

Generalized function from a business class

For more context.

//Code for generating work parameters in each batch of work
...
foreach (var workBatch in batchesOfWork)
{
  var tasks = workBatch.Select(async batch => 
    workBatch.Result = await GetData(/* work related parms*/)
  );
  await Task.WhenAll(tasks);
}
...

GetData() function

//code for formating url
try
{
  response = await HttpClientWrapper.CallAPI(formattedUrl, HttpMethod.Get, cts);
}
catch (TimeoutException)
{
  //retry logic
}
...
//JSON deserialization, error handling, etc.....

Edit

I forgot to mention that this also set on start-up.

   ServicePointManager
   .FindServicePoint(/* base uri for the API that we are contacting*/)
   .ConnectionLeaseTimeout = 60000; // 1 minute
    ServicePointManager.DnsRefreshTimeout = 60000;

Answer 1

The above mentioned code example shows that a common instance of HttpClient is being used by all the running applications.

Microsoft documentation recommends that the HttpClient object be instantiated once per application, rather than per-use.

This recommendation is applicable for the requests within one application. This is for the purpose of ensuring common connection settings for all requests made to specific destination API.

However, when there are multiple applications, then the recommended approach is to have one instance of HttpClient per application instance, in order to avoid the scenario of one application waiting for the other to finish.

Removing the static keyword for the HttpClientWrapper class and updating the code so that each application can have its own instance of HttpClient will resolve the reported problem.

More information: https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=netcore-3.1

Answer 2

After taking @David Browne - Microsoft's advice in the comment section, I changed the default amount of connections from the default (2) to the API provider's rate limit for my organization (100) and that seems to have done the trick. It has been several days since I've installed the change to production, and it is humming along nicely.

Additionally, I slimmed down the HttpClientWrapper class I had to contain the CallAPI function and a default HttpClientHandler implementation with the proxy/decompression settings I have above. It doesn't override the default timer anymore, as my thought is is that I should just retry the API call if it takes more than the default 100 seconds.

To anyone stumbling upon this thread:

1) One HttpClient being used throughout the entirety of your application will be fine, no matter the amount of threads or API calls being done by it. Just make sure to increase the number of DefaultConnections via the ServicePointManager . You also DO NOT have to use the HttpClient in a using context. It will work just fine in a lazy singleton as I demonstrate above. Don't worry about disposing of the HttpClient in a long running service.

2) Use async-await throughout your application. It is worth the pay-off as it makes the application much more readable and allows your threads to be freed up as you are awaiting a response back from the API. This might seem obvious, but it isn't if you haven't used the async-await architecture in an application before.