Correct way of using Task.WaitAll/WhenAll with batch execution and result returning functions

Question

I need to subdivide a group of NET.Core 3.1 HttpClient (Assembly System.Net.Http, Version=4.2.2.0) web requests in batches of 10, wait for these 10 call to complete, save the result somewhere, and repeat until all the batches are finished.

Example: I have an endpoint which receives an id and returns some json in return. I have a list with 25 user Id, I need to split the work in 10 web requests at time asyncrhonously, until completion, get the result and get to the next batch (so 10, get result, 10, get result, last 5, get result)

The best way to batch through Linq I found is using this code for batching in MoreLinq library , so in the code you will see the "Batch" method.

What I came up to was this:

//Pretend that HttpClient is already configured
//Concurrent queue to collect results from webservice, result order it's not i
var resultsCollection = new ConcurrentQueue<string>();
string inventedWebCall = @"http://fantasywebapi.com/searchById?";
//list of ids
var ids = Enumerable.Range(1, 25);

//Action that calls webrequest and enqueues result in thread safe queue
Action<int> webRequest = async (id) => {
    var webCall = $"{inventedWebCall}{id}";
    var webResult = await httpClient.GetAsync(new Uri(webCall));
    if (webResult.IsSuccessStatusCode)
    {
        resultsCollection.Enqueue(await webResult.Content.ReadAsStringAsync())
    }
};

//Gets a list of lists divided by 10 (10, 10, 5)
var batches = ids.Batch(10, batch => batch);
foreach (var batch in batches)
{
    //Collects the result of concurrent webrequest calls


    //Contains the tasks to await concurrently
    var batchTasks = new Task[batch.Count()];
    for (var i = 0; i < batch.Count(); i++)
    {
        batchTasks[i] = new Task(() => webRequest(batch.ElementAt(i)));
    }

    foreach (var batchToExec in batchTasks)
    {
        batchToExec.Start();
    }
    Task.WaitAll(batchTasks);
}

But it seems to have some trouble respecting the fact that the batches have to run in separate times, and populating the resulting queue.

Can anyone help?

Answer 1

It looks like Tasks that you are creating start another task and do not wait for their completion.
Once await httpClient.GetAsync is reached, your overall task will be completed, as long as it doesn't await result of webRequest action.
Change webRequest type to Func<int, Task> and wait for them.
Your sample code will look like:

//Pretend that HttpClient is already configured
//Concurrent queue to collect results from webservice, result order it's not important
var resultsCollection = new ConcurrentQueue<string>();
string inventedWebCall = @"http://fantasywebapi.com/searchById?";
//list of ids
var ids = Enumerable.Range(1, 25);

//Action that calls webrequest and enqueues result in thread safe queue
Func<int, Task> webRequest = async (id) => {
    var webCall = $"{inventedWebCall}{id}";
    var webResult = await httpClient.GetAsync(new Uri(webCall));
    if (webResult.IsSuccessStatusCode)
    {
        resultsCollection.Enqueue(await webResult.Content.ReadAsStringAsync())
    }
};

//Gets a list of lists divided by 10 (10, 10, 5)
var batches = ids.Batch(10, batch => batch);
foreach (var batch in batches)
{
    //Contains the tasks to await concurrently
    var batchTasks = new Task[batch.Count()];
    for (var i = 0; i < batch.Count(); i++)
    {
        batchTasks[i] = webRequest(batch.ElementAt(i));
    }

    Task.WaitAll(batchTasks);
}