How to use Task.WhenAny and implement retry

Question

I have a solution that creates multiple I/O based tasks and I'm using Task.WhenAny() to manage these tasks. But often many of the tasks will fail due to network issue or request throttling etc.

I can't seem to find a solution that enables me to successfully retry failed tasks when using a Task.WhenAny() approach.

Here is what I'm doing:

var tasks = new List<Task<MyType>>();
foreach(var item in someCollection)
{
   task.Add(GetSomethingAsync());
}
while (tasks.Count > 0)
{
   var child = await Task.WhenAny(tasks);
   tasks.Remove(child);
   ???
}

So the above structure works for completing tasks, but I haven't found a way to handle and retry failing tasks. The await Task.WhenAny throws an AggregateException rather than allowing me to inspect a task status. When In the exception handler I no longer have any way to retry the failed task.

Answer 1

I believe it would be easier to retry within the tasks, and then replace the Task.WhenAny -in-a-loop antipattern with Task.WhenAll

Eg, using Polly :

var tasks = new List<Task<MyType>>();
var policy = ...; // See Polly documentation
foreach(var item in someCollection)
  tasks.Add(policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

or, more succinctly:

var policy = ...; // See Polly documentation
var tasks = someCollection.Select(item => policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

Answer 2

If you don't want to use the Polly library for some reason, you could use the Retry method bellow. It accepts a task factory, and keeps creating and then awaiting a task until it completes successfully, or the maxAttempts have been reached:

public static async Task<TResult> Retry<TResult>(Func<Task<TResult>> taskFactory,
    int maxAttempts)
{
    int failedAttempts = 0;
    while (true)
    {
        try
        {
            var task = taskFactory();
            return await task.ConfigureAwait(false);
        }
        catch
        {
            failedAttempts++;
            if (failedAttempts >= maxAttempts) throw;
        }
    }
}

You could then use this method to download (for example) some web pages.

string[] urls =
{
    "https://stackoverflow.com",
    "https://superuser.com",
    //"https://no-such.url",
};
var httpClient = new HttpClient();
var tasks = urls.Select(url => Retry(async () =>
{
    return (Url: url, Html: await httpClient.GetStringAsync(url));
}, maxAttempts: 5));

var results = await Task.WhenAll(tasks);
foreach (var result in results)
{
    Console.WriteLine($"Url: {result.Url}, {result.Html.Length:#,0} chars");
}

Output:

Url: https://stackoverflow.com , 112,276 chars
Url: https://superuser.com , 122,784 chars

If you uncomment the third url then instead of these results an HttpRequestException will be thrown, after five failed attempts.

The Task.WhenAll method will wait for the completion of all tasks before propagating the error. In case it is preferable to report the error as soon as possible, you can find solutions in this question.