在 ASP.NET Core 3.1 中上传和下载大文件？

Question

我正在使用干净的架构开发 ASP.NET Core 3.1 API 项目，我有以下类库（层）：

• 基础设施（安全的东西和上传助手等...）
• 持久化（DA层）
• 领域（领域模型）
• 应用程序（用例 - 业务逻辑）
• API（API项目作为我的启动项目）

我希望能够将大文件上传到服务器（例如 2Gb 的文件大小甚至更多）并在之后下载它们，并且希望在以后不会出现 memory 溢出等问题。

任何帮助，将不胜感激。

Answer 1

如果您有那么大的文件，请不要在代码中使用byte[]或MemoryStream 。 仅当您下载/上传文件时才对流进行操作。

你有几个选择：

• 如果您同时控制客户端和服务器，请考虑使用类似tus的东西。 .NET 有客户端和服务器实现。 这可能是最简单和最强大的选择。
• 如果您使用 HttpClient 上传大文件，只需使用StreamContent class 发送它们。 同样，不要使用MemoryStream作为源，而是使用FileStream之类的其他东西。
• 如果您使用 HttpClient 下载大文件，请务必指定 HttpCompletionOptions，例如var response = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead) 。 否则，HttpClient 将缓冲 memory 中的整个响应。 然后，您可以通过var stream = response.Content.ReadAsStreamAsync()将响应文件作为 stream 处理。

ASP.NET 内核具体建议：

• 如果要通过 HTTP POST 接收文件，则需要增加请求大小限制： [RequestSizeLimit(10L * 1024L * 1024L * 1024L)]和[RequestFormLimits(MultipartBodyLengthLimit = 10L * 1024L * 1024L * 1024L)] 另外，需要禁用表单值绑定，否则整个请求会被缓冲到memory中：

   [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
   public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
   {
       public void OnResourceExecuting(ResourceExecutingContext context)
       {
           var factories = context.ValueProviderFactories;
           factories.RemoveType<FormValueProviderFactory>();
           factories.RemoveType<FormFileValueProviderFactory>();
           factories.RemoveType<JQueryFormValueProviderFactory>();
       }

       public void OnResourceExecuted(ResourceExecutedContext context)
       {
       }
   }

• 要从 controller 返回文件，只需通过File方法简单地返回它，该方法接受 stream： return File(stream, mimeType, fileName);

---

示例 controller 看起来像这样（请参阅https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1了解缺少的帮助程序类）：

private const MaxFileSize = 10L * 1024L * 1024L * 1024L; // 10GB, adjust to your need

[DisableFormValueModelBinding]
[RequestSizeLimit(MaxFileSize)]
[RequestFormLimits(MultipartBodyLengthLimit = MaxFileSize)]
public async Task ReceiveFile()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
        throw new BadRequestException("Not a multipart request");

    var boundary = MultipartRequestHelper.GetBoundary(MediaTypeHeaderValue.Parse(Request.ContentType));
    var reader = new MultipartReader(boundary, Request.Body);

    // note: this is for a single file, you could also process multiple files
    var section = await reader.ReadNextSectionAsync();

    if (section == null)
        throw new BadRequestException("No sections in multipart defined");

    if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition))
        throw new BadRequestException("No content disposition in multipart defined");

    var fileName = contentDisposition.FileNameStar.ToString();
    if (string.IsNullOrEmpty(fileName))
    {
        fileName = contentDisposition.FileName.ToString();
    }

    if (string.IsNullOrEmpty(fileName))
        throw new BadRequestException("No filename defined.");

    using var fileStream = section.Body;
    await SendFileSomewhere(fileStream);
}

// This should probably not be inside the controller class
private async Task SendFileSomewhere(Stream stream)
{
    using var request = new HttpRequestMessage()
    {
        Method = HttpMethod.Post,
        RequestUri = new Uri("YOUR_DESTINATION_URI"),
        Content = new StreamContent(stream),
    };
    using var response = await _httpClient.SendAsync(request);
    // TODO check response status etc.
}

在这个例子中，我们将 stream 整个文件发送给另一个服务。 在某些情况下，最好将文件临时保存到磁盘上。

Answer 2

我发现这篇文章很有用 - https://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient

这是那里提供的用于下载大文件的代码版本：

static public async Task HttpDownloadFileAsync(HttpClient httpClient, string url, string fileToWriteTo) {
  using HttpResponseMessage response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
  using Stream streamToReadFrom = await response.Content.ReadAsStreamAsync(); 
  using Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create); 
  await streamToReadFrom.CopyToAsync(streamToWriteTo);
}

Answer 3

有时问题是我们使用 Nginx 作为我们 asp.net 核心应用程序的前端代理，部署在 ubuntu/Linux 环境中的 docker 中。 这正是我尝试在 docker 或 .net 核心端调试的情况，但实际解决方案是通过将 Nginx 配置配置为

client_max_body_size 50M；

此行可以添加到您遇到问题的站点的 Nginx 配置的位置或服务器设置部分。

可能对某人有帮助。

Answer 4

问题是您必须处理长文件，无论您在哪里使用它们，您都需要大量资源才能阅读。 一种可能的解决方案是根据信息将文件划分为不同的块，或者在单独的作业或线程中处理它，或者使用 .net 中的并行性来处理它。 您可以指定文件的大小也请阅读以下对您非常有用的博客。

上传大文件

Answer 5

我建议阅读 @ManuelAllenspach 也链接到的 ASP.NET Core 中的 Microsoft Docs 上传文件。

https://learn.microsoft.com/en-us/as.net/core/mvc/models/file-uploads?view=as.netcore-6.0

根据您的要求，您应该查看的部分是使用流媒体上传大文件：（例如 2Gb 的文件大小或更多）

https://learn.microsoft.com/en-us/as.net/core/mvc/models/file-uploads?view=as.netcore-6.0#upload-large-files-with-streaming

public class GenerateAntiforgeryTokenCookieAttribute : ResultFilterAttribute
{
    public override void OnResultExecuting(ResultExecutingContext context)
    {
        var antiforgery = context.HttpContext.RequestServices.GetService<IAntiforgery>();

        // Send the request token as a JavaScript-readable cookie
        var tokens = antiforgery.GetAndStoreTokens(context.HttpContext);

        context.HttpContext.Response.Cookies.Append(
            "RequestVerificationToken",
            tokens.RequestToken,
            new CookieOptions() { HttpOnly = false });
    }

    public override void OnResultExecuted(ResultExecutedContext context)
    {
    }
}

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
    public void OnResourceExecuting(ResourceExecutingContext context)
    {
        var factories = context.ValueProviderFactories;
        factories.RemoveType<FormValueProviderFactory>();
        factories.RemoveType<FormFileValueProviderFactory>();
        factories.RemoveType<JQueryFormValueProviderFactory>();
    }

    public void OnResourceExecuted(ResourceExecutedContext context)
    {
    }
}

services.AddRazorPages(options =>
{
    options.Conventions
        .AddPageApplicationModelConvention("/StreamedSingleFileUploadDb",
            model =>
            {
                model.Filters.Add(
                    new GenerateAntiforgeryTokenCookieAttribute());
                model.Filters.Add(
                    new DisableFormValueModelBindingAttribute());
            });
    options.Conventions
        .AddPageApplicationModelConvention("/StreamedSingleFileUploadPhysical",
            model =>
            {
                model.Filters.Add(
                    new GenerateAntiforgeryTokenCookieAttribute());
                model.Filters.Add(
                    new DisableFormValueModelBindingAttribute());
            });
});

[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File", 
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

    // Accumulate the form data key-value pairs in the request (formAccumulator).
    var formAccumulator = new KeyValueAccumulator();
    var trustedFileNameForDisplay = string.Empty;
    var untrustedFileNameForStorage = string.Empty;
    var streamedFileContent = Array.Empty<byte>();

    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        _defaultFormOptions.MultipartBoundaryLengthLimit);
    var reader = new MultipartReader(boundary, HttpContext.Request.Body);

    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        var hasContentDispositionHeader = 
            ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

        if (hasContentDispositionHeader)
        {
            if (MultipartRequestHelper
                .HasFileContentDisposition(contentDisposition))
            {
                untrustedFileNameForStorage = contentDisposition.FileName.Value;
                // Don't trust the file name sent by the client. To display
                // the file name, HTML-encode the value.
                trustedFileNameForDisplay = WebUtility.HtmlEncode(
                        contentDisposition.FileName.Value);

                streamedFileContent = 
                    await FileHelpers.ProcessStreamedFile(section, contentDisposition, 
                        ModelState, _permittedExtensions, _fileSizeLimit);

                if (!ModelState.IsValid)
                {
                    return BadRequest(ModelState);
                }
            }
            else if (MultipartRequestHelper
                .HasFormDataContentDisposition(contentDisposition))
            {
                // Don't limit the key name length because the 
                // multipart headers length limit is already in effect.
                var key = HeaderUtilities
                    .RemoveQuotes(contentDisposition.Name).Value;
                var encoding = GetEncoding(section);

                if (encoding == null)
                {
                    ModelState.AddModelError("File", 
                        $"The request couldn't be processed (Error 2).");
                    // Log error

                    return BadRequest(ModelState);
                }

                using (var streamReader = new StreamReader(
                    section.Body,
                    encoding,
                    detectEncodingFromByteOrderMarks: true,
                    bufferSize: 1024,
                    leaveOpen: true))
                {
                    // The value length limit is enforced by 
                    // MultipartBodyLengthLimit
                    var value = await streamReader.ReadToEndAsync();

                    if (string.Equals(value, "undefined", 
                        StringComparison.OrdinalIgnoreCase))
                    {
                        value = string.Empty;
                    }

                    formAccumulator.Append(key, value);

                    if (formAccumulator.ValueCount > 
                        _defaultFormOptions.ValueCountLimit)
                    {
                        // Form key count limit of 
                        // _defaultFormOptions.ValueCountLimit 
                        // is exceeded.
                        ModelState.AddModelError("File", 
                            $"The request couldn't be processed (Error 3).");
                        // Log error

                        return BadRequest(ModelState);
                    }
                }
            }
        }

        // Drain any remaining section body that hasn't been consumed and
        // read the headers for the next section.
        section = await reader.ReadNextSectionAsync();
    }

    // Bind form data to the model
    var formData = new FormData();
    var formValueProvider = new FormValueProvider(
        BindingSource.Form,
        new FormCollection(formAccumulator.GetResults()),
        CultureInfo.CurrentCulture);
    var bindingSuccessful = await TryUpdateModelAsync(formData, prefix: "",
        valueProvider: formValueProvider);

    if (!bindingSuccessful)
    {
        ModelState.AddModelError("File", 
            "The request couldn't be processed (Error 5).");
        // Log error

        return BadRequest(ModelState);
    }

    // **WARNING!**
    // In the following example, the file is saved without
    // scanning the file's contents. In most production
    // scenarios, an anti-virus/anti-malware scanner API
    // is used on the file before making the file available
    // for download or for use by other systems. 
    // For more information, see the topic that accompanies 
    // this sample app.

    var file = new AppFile()
    {
        Content = streamedFileContent,
        UntrustedName = untrustedFileNameForStorage,
        Note = formData.Note,
        Size = streamedFileContent.Length, 
        UploadDT = DateTime.UtcNow
    };

    _context.File.Add(file);
    await _context.SaveChangesAsync();

    return Created(nameof(StreamingController), null);
}

using System;
using System.IO;
using Microsoft.Net.Http.Headers;

namespace SampleApp.Utilities
{
    public static class MultipartRequestHelper
    {
        // Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
        // The spec at https://tools.ietf.org/html/rfc2046#section-5.1 states that 70 characters is a reasonable limit.
        public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
        {
            var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;

            if (string.IsNullOrWhiteSpace(boundary))
            {
                throw new InvalidDataException("Missing content-type boundary.");
            }

            if (boundary.Length > lengthLimit)
            {
                throw new InvalidDataException(
                    $"Multipart boundary length limit {lengthLimit} exceeded.");
            }

            return boundary;
        }

        public static bool IsMultipartContentType(string contentType)
        {
            return !string.IsNullOrEmpty(contentType)
                   && contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
        }

        public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="key";
            return contentDisposition != null
                && contentDisposition.DispositionType.Equals("form-data")
                && string.IsNullOrEmpty(contentDisposition.FileName.Value)
                && string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
        }

        public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
            return contentDisposition != null
                && contentDisposition.DispositionType.Equals("form-data")
                && (!string.IsNullOrEmpty(contentDisposition.FileName.Value)
                    || !string.IsNullOrEmpty(contentDisposition.FileNameStar.Value));
        }
    }
}

[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadPhysical()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File", 
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        _defaultFormOptions.MultipartBoundaryLengthLimit);
    var reader = new MultipartReader(boundary, HttpContext.Request.Body);
    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        var hasContentDispositionHeader = 
            ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

        if (hasContentDispositionHeader)
        {
            // This check assumes that there's a file
            // present without form data. If form data
            // is present, this method immediately fails
            // and returns the model error.
            if (!MultipartRequestHelper
                .HasFileContentDisposition(contentDisposition))
            {
                ModelState.AddModelError("File", 
                    $"The request couldn't be processed (Error 2).");
                // Log error

                return BadRequest(ModelState);
            }
            else
            {
                // Don't trust the file name sent by the client. To display
                // the file name, HTML-encode the value.
                var trustedFileNameForDisplay = WebUtility.HtmlEncode(
                        contentDisposition.FileName.Value);
                var trustedFileNameForFileStorage = Path.GetRandomFileName();

                // **WARNING!**
                // In the following example, the file is saved without
                // scanning the file's contents. In most production
                // scenarios, an anti-virus/anti-malware scanner API
                // is used on the file before making the file available
                // for download or for use by other systems. 
                // For more information, see the topic that accompanies 
                // this sample.

                var streamedFileContent = await FileHelpers.ProcessStreamedFile(
                    section, contentDisposition, ModelState, 
                    _permittedExtensions, _fileSizeLimit);

                if (!ModelState.IsValid)
                {
                    return BadRequest(ModelState);
                }

                using (var targetStream = System.IO.File.Create(
                    Path.Combine(_targetFilePath, trustedFileNameForFileStorage)))
                {
                    await targetStream.WriteAsync(streamedFileContent);

                    _logger.LogInformation(
                        "Uploaded file '{TrustedFileNameForDisplay}' saved to " +
                        "'{TargetFilePath}' as {TrustedFileNameForFileStorage}", 
                        trustedFileNameForDisplay, _targetFilePath, 
                        trustedFileNameForFileStorage);
                }
            }
        }

        // Drain any remaining section body that hasn't been consumed and
        // read the headers for the next section.
        section = await reader.ReadNextSectionAsync();
    }

    return Created(nameof(StreamingController), null);
}