Consuming an IAsyncEnumerable

Question

I have one web API application that has a function that needs to stream some data. Then I have another web application that basically acts like a proxy for that. This other application does a whole lot of other stuff like handling authorization and such. It needs to connect to the API endpoint and pass the stream through potentially doing some addition processing for each row of data. In one case it will need to write out that data into an Excel file, for example.

Unfortunately, I see a lot of tutorials on how to create an IAsyncEnumerable, but not any about actually connecting to an endpoint that exposes one.

Ideally, I would like to actually wrap my IAsyncEnumerable and return it with some other stuff . This is what I basically do with other non-streamed endpoints already:

public class NonStreamedResult : BaseResult
{
    public SomeMetaData MetaData {get; set; }
    // here Data is usually small, so returning it all at once is no big deal
    public IEnumerable<SomeData> Data {get; set; }  
}

//....controller
public async Task<NonStreamedResult> GetNonStreamedData()

And on the consuming side, I'm using an HttpClient , awaiting an HttpResponseMessage and using ReadAsAsync<T> to turn that response into an object.

However, if I try to do this:

public class StreamedResult : BaseResult
{
    public SomeOtherMetaData MetaData {get; set; }
    // Here data could be huge, so we don't want to have to hold it all in memory at once
    public IAsyncEnumerable<SomeData> Data {get; set; }
}

//....controller
public async Task<StreamedResult> GetStreamedData()

I will get an error when it tries to deserialize because it doesn't know what concrete class to convert the IAsyncEnumerable into.

So what's the right way to connect to an endpoint that is using an IAsyncEnumerable without loading the entire response into memory (obviously) so that I can eventually return another IAsyncEnumerable from the proxy app?

It would be nice if I could still pass it wrapped, but it's not the end of the world if I have to work with just the naked IAsyncEnumerable

Note that I'm using the Microsoft.Bcl.AsyncInterfaces NuGet package since I'm still targeting an earlier version of the .NET framework.

Answer 1

I'm still targeting an earlier version of the .NET framework.

That's not going to work well. Unbuffered IAsyncEnumerable<T> support was added in ASP.NET 6 .

If you can't upgrade, then you'll need to write your own streaming JSON parser and your own streaming JSON serializer wrapped up in a custom formatter that flushes periodically. This is a non-trivial amount of work. The remainder of my answer assumes you will move to .NET 6.

It would be nice if I could still pass it wrapped, but it's not the end of the world if I have to work with just the naked IAsyncEnumerable

Wrapped can't possibly work. Just consider the situation where you define multiple IAsyncEnumerable<T> properties. When dealing with IAsyncEnumerable<T> in ASP.NET 6, you must return it as a top-level type. Metadata can be represented as headers.

To consume, first ensure that you set the HttpClient completion options so that the HTTP operation is considered "complete" when the headers are received, and then you can take the response body stream and pass it to JsonSerializer.DeserializeAsyncEnumerable . After retrieving each item from the downstream API, you can do whatever processing you need and then yield it to be streamed to your client.