How can I send HTTP GETs to several URLs using .NET async code and get the first result?

Question

I understand async javascript but aync .NET has a different approach and I still haven't got my head around it properly.

I have a list of URLs that I would like to check. I would like to check them asynchronously and get the first one that returns a certain status code. In this case I am looking for status code 401 (Unauthorized) as this indicates that it is a login challenge, which is what I am expecting. So I can't just use Task.WaitAny because I need to run some code to see which one matches my status code first.

Can anyone give me an example of how you run a callback on an aync task and then stop all the other tasks if you found what you want?

I am using .NET 4 in this project and would prefer to stick with this if possible. I have the System.Net.Http.HttpClient nuget package installed.

UPDATE: I have put together the following code, which I have finally got to produce the correct results, except I think it is waiting for each task - missing the whole point of being async. Not sure about use of new Task() or t.Wait() within the inner task but it seem the only way to catch the exception. (Exceptions happen on DNS fail and connection timeouts - I don't know a better way to handle those than catching and ignoring the exceptions.)

Any advice on improving this code to make it actually async?

    public async Task<ActionResult> Test() {
        //var patterns = GetPatterns();
        var patterns = "http://stackoverflow.com/,https://www.google.com,http://www.beweb.co.nz,https://outlook.office365.com/Microsoft-Server-ActiveSync,http://rubishnotexist.com".Split(",").ToList();

        var httpClient = new System.Net.Http.HttpClient();
        string result = "";
        CancellationTokenSource source = new CancellationTokenSource();
        CancellationToken cancellationToken = source.Token;
        var allTasks = new List<Task>();
        foreach (var pattern in patterns) {
            var url = pattern;

            Task task = new Task(() => {
                string answer = "";
                var st = DateTime.Now;
                var t = httpClient.GetAsync(pattern, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
                t.ContinueWith(d => {
                    if (!source.IsCancellationRequested) {
                        if (t.IsFaulted) {
                            answer = "Fault - " + " " + url;
                        } else if (d.Result.StatusCode == System.Net.HttpStatusCode.Unauthorized) {
                            // found it - so cancel all others
                            answer = "YES - " + d.Result.StatusCode + " " + url;
                            //source.Cancel();
                        } else {
                            answer = "No - " + d.Result.StatusCode + " " + url;
                        }
                    }
                    result += answer + " ("+(DateTime.Now-st).TotalMilliseconds+"ms)<br>";
                });
                try {
                    t.Wait();
                } catch (Exception) {
                    // ignore eg DNS fail and connection timeouts
                }
            });

            allTasks.Add(task);
            task.Start();
        }

        // Wait asynchronously for all of them to finish
        Task.WaitAll(allTasks.ToArray());

        return Content(result + "<br>DONE");
    }

In the above I didn't have the cancellation part working. Here is a version including cancellation:

    public async Task<ActionResult> Test2(string email) {
        var patterns = GetPatterns(email);
        patterns = "http://stackoverflow.com/,https://www.google.com,http://www.beweb.co.nz,https://outlook.office365.com/Microsoft-Server-ActiveSync,http://rubishnotexist.com".Split(",").ToList();
        var httpClient = new System.Net.Http.HttpClient();

        string result = "";
        CancellationTokenSource source = new CancellationTokenSource();
        CancellationToken cancellationToken = source.Token;
        var allTasks = new List<Task>();
        foreach (var pattern in patterns) {
            var url = pattern;

            Task task = new Task(() => {
                string answer = "";
                var st = DateTime.Now;
                var t = httpClient.GetAsync(pattern, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
                t.ContinueWith(d => {
                    if (!source.IsCancellationRequested) {
                        if (t.IsFaulted) {
                            answer = "Fault - " + " " + url;
                        } else if (d.Result.StatusCode == System.Net.HttpStatusCode.Unauthorized) {
                            // found it - so cancel all others
                            answer = "YES - " + d.Result.StatusCode + " " + url;
                            result += answer + " (" + (DateTime.Now - st).TotalMilliseconds + "ms)  <-- cancelled here <br>";
                            source.Cancel();
                        } else {
                            answer = "No - " + d.Result.StatusCode + " " + url;
                        }
                    } else {
                            answer = "cancelled - " + url;
                    }
                    result += answer + " (" + (DateTime.Now - st).TotalMilliseconds + "ms)<br>";
                });
                try {
                    t.Wait();
                } catch (Exception) {
                    // ignore
                }
            });

            allTasks.Add(task);
            task.Start();
        }

        // Wait asynchronously for all of them to finish
        Task.WaitAll(allTasks.ToArray());

        return Content(result + "<br>DONE");
    }

Answer 1

Use Task.WhenAll() instead, then examine the results of the tasks.

To prevent other tasks continuing once any one throws an exception, you can create a single CancellationToken (by first creating a CancellationTokenSource , then using its .Token ) that you pass to all the tasks, and on failure, you cancel the token; see How to cancel and raise an exception on Task.WhenAll if any exception is raised? for more details and sample code. All the tasks then observe the token, and optionally explicitly check it occasionally and exit if it's canceled. They should also pass it on to those methods that support it, so they, in turn, can cancel quickly when the token is canceled.

Re exceptions, this answer covers them pretty well. If you want no exception thrown into the calling code, you should handle the exception within each task create instead, but then you will need to modify the above canceling mechanism accordingly. You could instead just catch the single exception that await Task.WhenAll() might throw and at that point observe all the exceptions thrown in the Task.Exception property of each task, or ignore them if that is the desired result.

---

Re canceling on success (from the comments) - I guess there are many ways to do it, but one could be:

using (var cts = new CancellationTokenSource())
{
    var tasks = new List<Task<HttpStatusCode>>();

    foreach (var url in patterns)
    {
        tasks.Add(GetStatusCodeAsync(url, cts.Token));
    }

    while (tasks.Any() && !cts.IsCancellationRequested)
    {
        Task<HttpStatusCode> task = await Task.WhenAny(tasks);

        if (await task == HttpStatusCode.Unauthorized)
        {
            cts.Cancel();
            // Handle the "found" situation
            // ...
        }
        else
        {
            tasks.Remove(task);
        }
    }
}

and then put your HttpClient code in a separate method:

private static async Task<HttpStatusCode> GetStatusCodeAsync(object url, CancellationToken token)
{
    try
    {
        // Your HttpClient code
        // ...
        await <things>;
        // (pass token on to methods that support it)
        // ...
        return httpStatusCode;
    }
    catch (Exception e)
    {
        // Don't rethrow if you handle everything here
        return HttpStatusCode.Unused; // (or whatever)
    }
}