[英]Uploading and Downloading large files in ASP.NET Core 3.1?
我正在使用干凈的架構開發 ASP.NET Core 3.1 API 項目,我有以下類庫(層):
我希望能夠將大文件上傳到服務器(例如 2Gb 的文件大小甚至更多)並在之后下載它們,並且希望在以后不會出現 memory 溢出等問題。
任何幫助,將不勝感激。
如果您有那么大的文件,請不要在代碼中使用byte[]
或MemoryStream
。 僅當您下載/上傳文件時才對流進行操作。
你有幾個選擇:
StreamContent
class 發送它們。 同樣,不要使用MemoryStream
作為源,而是使用FileStream
之類的其他東西。var response = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead)
。 否則,HttpClient 將緩沖 memory 中的整個響應。 然后,您可以通過var stream = response.Content.ReadAsStreamAsync()
將響應文件作為 stream 處理。ASP.NET 內核具體建議:
[RequestSizeLimit(10L * 1024L * 1024L * 1024L)]
和[RequestFormLimits(MultipartBodyLengthLimit = 10L * 1024L * 1024L * 1024L)]
另外,需要禁用表單值綁定,否則整個請求會被緩沖到memory中: [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var factories = context.ValueProviderFactories;
factories.RemoveType<FormValueProviderFactory>();
factories.RemoveType<FormFileValueProviderFactory>();
factories.RemoveType<JQueryFormValueProviderFactory>();
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}
File
方法簡單地返回它,該方法接受 stream: return File(stream, mimeType, fileName);
示例 controller 看起來像這樣(請參閱https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1了解缺少的幫助程序類):
private const MaxFileSize = 10L * 1024L * 1024L * 1024L; // 10GB, adjust to your need
[DisableFormValueModelBinding]
[RequestSizeLimit(MaxFileSize)]
[RequestFormLimits(MultipartBodyLengthLimit = MaxFileSize)]
public async Task ReceiveFile()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
throw new BadRequestException("Not a multipart request");
var boundary = MultipartRequestHelper.GetBoundary(MediaTypeHeaderValue.Parse(Request.ContentType));
var reader = new MultipartReader(boundary, Request.Body);
// note: this is for a single file, you could also process multiple files
var section = await reader.ReadNextSectionAsync();
if (section == null)
throw new BadRequestException("No sections in multipart defined");
if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition))
throw new BadRequestException("No content disposition in multipart defined");
var fileName = contentDisposition.FileNameStar.ToString();
if (string.IsNullOrEmpty(fileName))
{
fileName = contentDisposition.FileName.ToString();
}
if (string.IsNullOrEmpty(fileName))
throw new BadRequestException("No filename defined.");
using var fileStream = section.Body;
await SendFileSomewhere(fileStream);
}
// This should probably not be inside the controller class
private async Task SendFileSomewhere(Stream stream)
{
using var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("YOUR_DESTINATION_URI"),
Content = new StreamContent(stream),
};
using var response = await _httpClient.SendAsync(request);
// TODO check response status etc.
}
在這個例子中,我們將 stream 整個文件發送給另一個服務。 在某些情況下,最好將文件臨時保存到磁盤上。
我發現這篇文章很有用 - https://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient
這是那里提供的用於下載大文件的代碼版本:
static public async Task HttpDownloadFileAsync(HttpClient httpClient, string url, string fileToWriteTo) {
using HttpResponseMessage response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
using Stream streamToReadFrom = await response.Content.ReadAsStreamAsync();
using Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create);
await streamToReadFrom.CopyToAsync(streamToWriteTo);
}
有時問題是我們使用 Nginx 作為我們 asp.net 核心應用程序的前端代理,部署在 ubuntu/Linux 環境中的 docker 中。 這正是我嘗試在 docker 或 .net 核心端調試的情況,但實際解決方案是通過將 Nginx 配置配置為
client_max_body_size 50M;
此行可以添加到您遇到問題的站點的 Nginx 配置的位置或服務器設置部分。
可能對某人有幫助。
問題是您必須處理長文件,無論您在哪里使用它們,您都需要大量資源才能閱讀。 一種可能的解決方案是根據信息將文件划分為不同的塊,或者在單獨的作業或線程中處理它,或者使用 .net 中的並行性來處理它。 您可以指定文件的大小也請閱讀以下對您非常有用的博客。
我建議閱讀 @ManuelAllenspach 也鏈接到的 ASP.NET Core 中的 Microsoft Docs 上傳文件。
https://learn.microsoft.com/en-us/as.net/core/mvc/models/file-uploads?view=as.netcore-6.0
根據您的要求,您應該查看的部分是使用流媒體上傳大文件:(例如 2Gb 的文件大小或更多)
public class GenerateAntiforgeryTokenCookieAttribute : ResultFilterAttribute
{
public override void OnResultExecuting(ResultExecutingContext context)
{
var antiforgery = context.HttpContext.RequestServices.GetService<IAntiforgery>();
// Send the request token as a JavaScript-readable cookie
var tokens = antiforgery.GetAndStoreTokens(context.HttpContext);
context.HttpContext.Response.Cookies.Append(
"RequestVerificationToken",
tokens.RequestToken,
new CookieOptions() { HttpOnly = false });
}
public override void OnResultExecuted(ResultExecutedContext context)
{
}
}
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var factories = context.ValueProviderFactories;
factories.RemoveType<FormValueProviderFactory>();
factories.RemoveType<FormFileValueProviderFactory>();
factories.RemoveType<JQueryFormValueProviderFactory>();
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}
services.AddRazorPages(options =>
{
options.Conventions
.AddPageApplicationModelConvention("/StreamedSingleFileUploadDb",
model =>
{
model.Filters.Add(
new GenerateAntiforgeryTokenCookieAttribute());
model.Filters.Add(
new DisableFormValueModelBindingAttribute());
});
options.Conventions
.AddPageApplicationModelConvention("/StreamedSingleFileUploadPhysical",
model =>
{
model.Filters.Add(
new GenerateAntiforgeryTokenCookieAttribute());
model.Filters.Add(
new DisableFormValueModelBindingAttribute());
});
});
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 1).");
// Log error
return BadRequest(ModelState);
}
// Accumulate the form data key-value pairs in the request (formAccumulator).
var formAccumulator = new KeyValueAccumulator();
var trustedFileNameForDisplay = string.Empty;
var untrustedFileNameForStorage = string.Empty;
var streamedFileContent = Array.Empty<byte>();
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, HttpContext.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader =
ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
if (MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
untrustedFileNameForStorage = contentDisposition.FileName.Value;
// Don't trust the file name sent by the client. To display
// the file name, HTML-encode the value.
trustedFileNameForDisplay = WebUtility.HtmlEncode(
contentDisposition.FileName.Value);
streamedFileContent =
await FileHelpers.ProcessStreamedFile(section, contentDisposition,
ModelState, _permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
}
else if (MultipartRequestHelper
.HasFormDataContentDisposition(contentDisposition))
{
// Don't limit the key name length because the
// multipart headers length limit is already in effect.
var key = HeaderUtilities
.RemoveQuotes(contentDisposition.Name).Value;
var encoding = GetEncoding(section);
if (encoding == null)
{
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 2).");
// Log error
return BadRequest(ModelState);
}
using (var streamReader = new StreamReader(
section.Body,
encoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1024,
leaveOpen: true))
{
// The value length limit is enforced by
// MultipartBodyLengthLimit
var value = await streamReader.ReadToEndAsync();
if (string.Equals(value, "undefined",
StringComparison.OrdinalIgnoreCase))
{
value = string.Empty;
}
formAccumulator.Append(key, value);
if (formAccumulator.ValueCount >
_defaultFormOptions.ValueCountLimit)
{
// Form key count limit of
// _defaultFormOptions.ValueCountLimit
// is exceeded.
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 3).");
// Log error
return BadRequest(ModelState);
}
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
// Bind form data to the model
var formData = new FormData();
var formValueProvider = new FormValueProvider(
BindingSource.Form,
new FormCollection(formAccumulator.GetResults()),
CultureInfo.CurrentCulture);
var bindingSuccessful = await TryUpdateModelAsync(formData, prefix: "",
valueProvider: formValueProvider);
if (!bindingSuccessful)
{
ModelState.AddModelError("File",
"The request couldn't be processed (Error 5).");
// Log error
return BadRequest(ModelState);
}
// **WARNING!**
// In the following example, the file is saved without
// scanning the file's contents. In most production
// scenarios, an anti-virus/anti-malware scanner API
// is used on the file before making the file available
// for download or for use by other systems.
// For more information, see the topic that accompanies
// this sample app.
var file = new AppFile()
{
Content = streamedFileContent,
UntrustedName = untrustedFileNameForStorage,
Note = formData.Note,
Size = streamedFileContent.Length,
UploadDT = DateTime.UtcNow
};
_context.File.Add(file);
await _context.SaveChangesAsync();
return Created(nameof(StreamingController), null);
}
using System;
using System.IO;
using Microsoft.Net.Http.Headers;
namespace SampleApp.Utilities
{
public static class MultipartRequestHelper
{
// Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
// The spec at https://tools.ietf.org/html/rfc2046#section-5.1 states that 70 characters is a reasonable limit.
public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
{
var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;
if (string.IsNullOrWhiteSpace(boundary))
{
throw new InvalidDataException("Missing content-type boundary.");
}
if (boundary.Length > lengthLimit)
{
throw new InvalidDataException(
$"Multipart boundary length limit {lengthLimit} exceeded.");
}
return boundary;
}
public static bool IsMultipartContentType(string contentType)
{
return !string.IsNullOrEmpty(contentType)
&& contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
}
public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="key";
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& string.IsNullOrEmpty(contentDisposition.FileName.Value)
&& string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
}
public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& (!string.IsNullOrEmpty(contentDisposition.FileName.Value)
|| !string.IsNullOrEmpty(contentDisposition.FileNameStar.Value));
}
}
}
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadPhysical()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 1).");
// Log error
return BadRequest(ModelState);
}
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, HttpContext.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader =
ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
// This check assumes that there's a file
// present without form data. If form data
// is present, this method immediately fails
// and returns the model error.
if (!MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 2).");
// Log error
return BadRequest(ModelState);
}
else
{
// Don't trust the file name sent by the client. To display
// the file name, HTML-encode the value.
var trustedFileNameForDisplay = WebUtility.HtmlEncode(
contentDisposition.FileName.Value);
var trustedFileNameForFileStorage = Path.GetRandomFileName();
// **WARNING!**
// In the following example, the file is saved without
// scanning the file's contents. In most production
// scenarios, an anti-virus/anti-malware scanner API
// is used on the file before making the file available
// for download or for use by other systems.
// For more information, see the topic that accompanies
// this sample.
var streamedFileContent = await FileHelpers.ProcessStreamedFile(
section, contentDisposition, ModelState,
_permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
using (var targetStream = System.IO.File.Create(
Path.Combine(_targetFilePath, trustedFileNameForFileStorage)))
{
await targetStream.WriteAsync(streamedFileContent);
_logger.LogInformation(
"Uploaded file '{TrustedFileNameForDisplay}' saved to " +
"'{TargetFilePath}' as {TrustedFileNameForFileStorage}",
trustedFileNameForDisplay, _targetFilePath,
trustedFileNameForFileStorage);
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
return Created(nameof(StreamingController), null);
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.