C# Stream Response from 3rd party, minimal buffering

Question

Our ASP.NET MVC endpoint is a behaving as a proxy to another 3rd party HTTP endpoint, which returns about 400MB of XML document generated dynamically.

Is there a way for ASP.NET MVC to "stream" that 3rd party response straight to the user of our endpoint with "minimal" buffering ?

At the moment, it looks like ASP.NET System.Web.Mvc.Controller.File() loads the whole file into memory as the response. Not sure how I can confirm this, other than the jump in memory usage ?

System.Web.Mvc.Controller.File(

The IIS AppPool memory usage increases by 400MB, which is then re-claimed by Garbage Collection later.

It will be nice if we can avoid System.Web.Mvc.Controller.File() loading the whole 400MB strings into memory, by streaming it "almost directly" from incoming response, is it possible ?

The mock c# linqpad code is roughly like this

public class MyResponseItem {
    public Stream myStream;
    public string metadata;
}

void Main()
{
    Stream stream = MyEndPoint();   

    //now let user download this XML as System.Web.Mvc.FileResult
    System.Web.Mvc.ActionResult fileResult = System.Web.Mvc.Controller.File(stream, "text/xml");
    fileResult.Dump();
}

Stream MyEndPoint() {
    MyResponseItem myResponse = GetStreamFromThirdParty("https://www.google.com");
    return myResponse.myStream;
}

MyResponseItem GetStreamFromThirdParty(string fullUrl)
{   
    MyResponseItem myResponse = new MyResponseItem();   
    System.Net.WebResponse webResponse = System.Net.WebRequest.Create(fullUrl).GetResponse();
    myResponse.myStream = webResponse.GetResponseStream();
    return myResponse;
}

Answer 1

You can reduce the memory footprint by not buffering and just copying the stream directly to output stream, an quick n' dirty example of this here:

    public async Task<ActionResult> Download()
    {
        using (var httpClient = new System.Net.Http.HttpClient())
        {
            using (
                var stream = await httpClient.GetStreamAsync(
                    "https://ckannet-storage.commondatastorage.googleapis.com/2012-10-22T184507/aft4.tsv.gz"
                    ))
            {
                Response.ContentType = "application/octet-stream";
                Response.Buffer = false;
                Response.BufferOutput = false;
                await stream.CopyToAsync(Response.OutputStream);
            }
            return new HttpStatusCodeResult(200);
        }
    }

If you want to reduce the footprint even more you can set a lower buffer size with the CopyToAsync(Stream, Int32) overload, default is 81920 bytes.

Answer 2

My requirement on proxy download also need to ensure the source ContentType (or any Header you need) can be forwarded as well. (eg If I proxy-download a video in http://techslides.com/sample-webm-ogg-and-mp4-video-files-for-html5 , I need to let user see the same browser-video-player screen as they open the link directly, but not jumping to file-download / hard-coded ContentType)

Basing on answer by @devlead + another post https://stackoverflow.com/a/30164356/4684232 , I adjusted a lil on the answer to fulfill my need. Here's my adjusted code in case anyone has the same need.

public async Task<ActionResult> Download(string url)
{
    using (var httpClient = new System.Net.Http.HttpClient())
    {
        using (var response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
        {
            response.EnsureSuccessStatusCode();
            using (var stream = await response.Content.ReadAsStreamAsync())
            {
                Response.ContentType = response.Content.Headers.ContentType.ToString();
                Response.Buffer = false;
                Response.BufferOutput = false;
                await stream.CopyToAsync(Response.OutputStream);
            }
        }

        return new HttpStatusCodeResult(200);
    }
}

ps HttpCompletionOption.ResponseHeadersRead is the important performance key. Without it, GetAsync will await until the whole source response stream are downloaded, which is much slower.