What is the fastest and safest way to send large number of requests with HttpClient?

Question

Initially I used the following to send out large number of URLs:

for (...) {
  using (var client = new HttpClient()) {
    client.GetAsync(url);
  } 
}

Turns out, that is the wrong way to use HttpClient, because .Dispose leaves the connection in place on the other side thereby wasting server resources. The suggested fix to this is to use a shared copy of httpClient:

var client = new HttpClient();
for (...) {
  client.GetAsync(url);
}

This approach suffers from an even worse problem. If your target url is behind a load balancer, httpClient establishes a connection with the first box it's routed to and every subsequent hit will go to that box ignoring the rest of the web farm.

So what is the correct way to use HttpClient in a way that doesn't waste resources on the other end of the connection and is able to utilize the entire web farm?

Answer 1

If you want use multiple HTTP connections, a straightforward solution would be something like this:

const int size = 12;
var clients = new HttpClient[size];
for (var i = 0; i < size; i++)
{
    clients[i] = new HttpClient();
}

int j = 0;
for (...)
{
    clients[j++%size].GetAsync(url); // todo: handle async correctly
}

If you are inside ASP.NET Core application, I would urge you to consider using IHttpClientFactory .

If you are not using IHttpClientFactory be aware that if DNS change occurs during the lifetime of your application, it won't be reflected.