HttpClient not writing to stream while downloading

Question

Currently I am implementing a way to report Progress with the HttpClient, since we share code with a .NET4 WPF and a Windows Universal App we use the Microsoft HTTP Client Libraries from NuGet. The idea was to wrap the target file stream in a CountingInputStream and report progress there:

 public override void Write(byte[] buffer, int offset, int count)
    {
        _stream.Write(buffer, offset, count);

        _bytesRead += count;
        _progress.Report(_bytesRead);

        if (_cancellationToken.IsCancellationRequested)
        {
            _cancellationToken.ThrowIfCancellationRequested();
        }
    }

Then I send my request with: HttpResponseMessage httpResponseMessage = AsyncHelpers.RunSync(() => _httpClient.SendAsync(httpRequestMessage, HttpCompletionOption.ResponseHeadersRead, cancellationToken));

After that I open the file stream and then copy the content stream. The response has correct headers: Content-Length: 213334 Content-Type: application/octet-stream; charset=UTF-8 Content-Type: application/octet-stream; charset=UTF-8 Content-Disposition: attachment; filename="Bondi Beach.jpg"; filename*=UTF-8''Bondi%20Beach.jpg Content-Disposition: attachment; filename="Bondi Beach.jpg"; filename*=UTF-8''Bondi%20Beach.jpg

using(Stream fileStream = new CountingInputStream(storage.Open(downloadRequest.TargetPath, FileMode.Create), downloadRequest.Progress, cancellationToken )) {                                               
                await HttpHeaderResponseMessage.Content.CopyToAsync(fileStream);
       }

The problem is that the StreamContent only starts writing to the file stream after the download has finished. When it started writing progress reporting just works fine.

I already tried different approaches like:

• ReadAsStreamAsync and then copy the response stream to file stream
• ReadAsStreamAsync manually read to buffer and then write to file stream
• _httpClient = new System.Net.Http.HttpClient(){MaxResponseContentBufferSize = 4096}; to restrict the BufferSize

Any ideas how I could force the ContentStream to write to the file stream while it is still downloading?

UPDATE: Following Luaans advice I tried to override the WriteAsync and implemented a StreamContent Extensions method:

//CountingInputStream
public Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
    {
        _bytesRead += count;
        _progress.Report(_bytesRead);

        if (_cancellationToken.IsCancellationRequested)
        {
            _cancellationToken.ThrowIfCancellationRequested();
        }
        return _stream.WriteAsync(buffer, offset, count, cancellationToken);
    }
//static Extensions Class
public static async Task CopyToAs(this StreamContent source, Stream targetStream)
    {
        int read;
        byte[] buffer = new byte[4096];
        using(Stream responseStream = await source.ReadAsStreamAsync()) {
            while ((read = await responseStream.ReadAsync(buffer,0,buffer.Length))>0) {
                await targetStream.WriteAsync(buffer, 0, read);
            }
        }
    }

It still waits till the download is finish until it calls ReadAsync the first time. Any hints what I have done wrong?

Answer 1

The fact that ReadAsStreamAsync is, well, async , makes this rather suspicious. Why would you asynchronously wait to get the stream ? You're supposed to read it asynchronously, but you should have the stream itself ready right away.

Reading the documentation makes this blatantly obvious:

This operation will not block. The returned task object will complete after the whole response (including content) is read.

However, there's overloads you can use to have it return after the headers have been read. This still means you need to wait for the server to process the request, before starting to get progress, but for the download itself, you are in luck.

Sample code:

var response = 
 await 
 (
   new HttpClient()
   .GetAsync("http://www.microsoft.com/", HttpCompletionOption.ResponseHeadersRead)
 );

var stream = await response.Content.ReadAsStreamAsync();
var buffer = new byte[2048];    

while (await stream.ReadAsync(buffer, 0, buffer.Length) > 0)
{
  // Report progress and write to a different stream
}

EDIT:

It sounds like you should be using Windows.Web.Http.HttpClient instead of System.Net.Http.HttpClient :

async Task DownloadWithProgress()
{
 var awaitable = httpClient.GetAsync(yourUrl)

 awaitable.Progress = (res, progress) =>
 {
   // Report progress
 }

 await awaitable;   
}