Await a task list without blocking another task list

Question

I'm working on a piece of code that takes data from one source, does some processing, then saves it to a different source. Due to a high level of latency on both ends, I'm trying to use async/await to wait for both the load and save operations to complete. Ideally this would start all of the input tasks, and when each one completes, it does some quick processing then starts the output task - using Task.WaitAll would result in the program doing nothing when the new tasks could be starting instead.

Since the second asynchronous task depends on the results of the first one, I can't figure out how to handle the second await in a way that lets it resume processing other results from the first await - once the result is processed and the second await is hit, it blocks completely until the save function is done instead of resuming to process other results. I'm trying something like this, what am I doing wrong?

async Task MainAsync()
{
    /* gets a list of id's */

    var dataRequests = ids.Select(id => LoadEntryById(id)).ToList();

    foreach(var request in dataRequests)
    {
        RawEntry response = await request;

        ProcessedEntry result = doSomething(response);

        await SaveResult(result);
    }
}

async Task<RawEntry> LoadEntryById(int id)
{
    /* IO task */
}

async Task SaveResult(ProcessedEntry result)
{
    /* IO task */
}

Answer 1

Instead of awaiting on single methods, create your List<Task> using your continuation and then use Task.WhenAll :

var tasks = ids.Select(async id => {
    var response = await LoadEntryById(id);
    var result = doSomething(response);
    await SaveResult(result);
}).ToList();

await Task.WhenAll(tasks);

This will process all your task in an asynchronous manner.

Answer 2

If i understand you correctly, you could try this:

Parallel.Foreach( ids.Select(id => LoadEntryById(id)), (rawEntry) => {
    var result = doSomething(rawEntry);

    await SaveResult(result);
}

RawEntry LoadEntryById(int id)
{
    /* IO task */
}

async Task SaveResult(ProcessedEntry result)
{
    /* IO task */
}

In this setup LoadEntryById do not need to return Task, but it depends whats it is doing.