Stream 通过 rest 端点将文件压缩到客户端

Question

I am trying to stream on the fly zipped files but memory consumption is high. For example, to zip total file size of 2.8 GB is taking nearly 5 GB of processor memory.

[Route("zip")]    
public class ZipController : ControllerBase
{
    private readonly HttpClient _httpClient;
    public ZipController()
    {
        _httpClient = new HttpClient();
    }

    [HttpPost]
    public async Task Zip([FromBody] JsonToZipInput input)
    {        

        Response.ContentType = "application/octet-stream";
        Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");
    
        using var zipArchive =
            new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
        foreach (var (key, value) in input.FilePathsToUrls)
        {
            var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
            await using var zipStream = zipEntry.Open();
            await using var stream = await _httpClient.GetStreamAsync(value);
            await stream.CopyToAsync(zipStream);
        }

    }

}

Answer 1

I believe you should be able to call Response.StartAsync :

[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{        
  Response.ContentType = "application/octet-stream";
  Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");

  await Response.StartAsync();
    
  using var zipArchive = new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
  foreach (var (key, value) in input.FilePathsToUrls)
  {
    var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
    await using var zipStream = zipEntry.Open();
    await using var stream = await _httpClient.GetStreamAsync(value);
    await stream.CopyToAsync(zipStream);
  }
}

StartAsync should start the response being sent. Note that neither the response headers nor the status code can be modified once StartAsync is called.

In particular, this means that your exception handling will be different. Previously, an exception (eg, from a bad URL in the request) would cause an exceptional status code (ie, 500). With a streaming response, any exceptions after StartAsync cannot change the status code; it's already been sent. Instead, it will appear to the client as though the connection was terminated without a clean close. Complicating this a bit further, this behavior is not uncommon for web servers to do in the successful case, so clients may not complain - they would just end up with truncated (invalid) zip files. (In the case of streaming zips, the "file table" in the zip is sent last instead of first).

So, this should work, but I also recommend:

1. Ensure your exception logging works for exceptions after StartAsync . There is no way to return error details to the client, so you must rely on logging.
2. If you control the client, test out this new error situation, and see if you can detect it. If it's not detectable using that client, then ensure your code validates the zip.