ASP.NET Core web application - How to upload large files

Question

Problem

I'm trying to create an ASP.NET Core (3.1) web application that accepts file uploads and then breaks it into chunks to send to Sharepoint via MS Graph API. There are a few other posts here that address the similar questions but they assume a certain level of .NET knowledge that I don't have just yet. So I'm hoping someone can help me cobble something together.

Configure Web server & app to Accept Large Files

I have done the following to allow IIS Express to upload up to 2GB files:

a) created a web.config file with the following code:

<?xml version="1.0" encoding="utf-8"?>
<configuration>

    <location path="Home/UploadFile">
        <system.webServer>
            <handlers>
                <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
            </handlers>
            <security>
                <requestFiltering>
                    <!--unit is bytes => 2GB-->
                    <requestLimits maxAllowedContentLength="2147483647" />
                </requestFiltering>
            </security>
        </system.webServer>
    </location>
</configuration>

B) I have the following in my Startup.cs Configuration section:

        //Add support for uploading large files  TODO:  DO I NEED THIS?????
        services.Configure<FormOptions>(x =>
        {

            x.ValueLengthLimit = int.MaxValue; // Limit on individual form values
            x.MultipartBodyLengthLimit = int.MaxValue; // Limit on form body size
            x.MultipartHeadersLengthLimit = int.MaxValue; // Limit on form header size
        });

        services.Configure<IISServerOptions>(options =>
        {
            options.MaxRequestBodySize = int.MaxValue;  //2GB
         });

Here's what my form looks like that allows the user to pick the file and submit:

@{
    ViewData["Title"] = "Messages";
}
<h1>@ViewData["Title"]</h1>

<p></p>
<form id="uploadForm" action="UploadFile" method="post" enctype="multipart/form-data">
    <dl>
        <dt>
            <label for="file">File</label>
        </dt>
        <dd>
            <input id="file" type="file" name="file" />
        </dd>
    </dl>

    <input class="btn" type="submit" value="Upload" />

    <div style="margin-top:15px">
        <output form="uploadForm" name="result"></output>
    </div>
</form>

Here's what the controller looks like:

    [HttpPost]
    [RequestSizeLimit(2147483647)]       //unit is bytes => 2GB
    [RequestFormLimits(MultipartBodyLengthLimit = 2147483647)]
    public async void UploadFile()
    {
        User currentUser = null;
        currentUser = await _graphServiceClient.Me.Request().GetAsync();
        //nothing have to do with the file has been written yet. 

    }

When the user clicks on the file button and chooses a large file, I no longer get IIS 413 error messages. Great. The logic hits the right method in my controller.

But I have the following questions for this part of the code:

• When the user picks the file... what is actually happening under the hood? Has the file actually been stuffed into my form and is accessible from my controller?

• Is it a stream?

• how do i get to the file?

• If ultimately, I need to send this file to Sharepoint using this type of an approach (the last example on chunking), it seems that the best approach is to save the file on my server somewhere... and then copy the sample code and try to chunk it out? The sample code seems to be referring to file paths and file sizes, I'm assuming I need to persist it to my web server somewhere first, and then take it from there.

• if i do need to save it, can you point me in the right direction - maybe some sample code that shows me how to take the POSTed data in my form and save it?

• ultimately, this will need to be refactored os that there is not GUI... but it's just an API that accepts large files to upload somewhere. But I think i'll try to learn how to do it this way first... and then refactor to change my code to be API only.

Sorry for the noob questions. I have tried to do my research before posting here. But somethings are still a bit fuzzy.

EDIT 1

Per the suggestion in one of the posted answers, i've downloaded sample code that demonstrates how to bypass saving to a local file on the web server. It's based on this article

I have created a web.config file again -to avoid the 413 errors from IIS. I have also edited the list of allowed file extensions to support.pdf and.docx and.mp4.

When I try to run the sample project, and I choose the "Stream a file with AJAX to a controller endpoint" under the "Physical Storage Upload Examples" section, it dies here:

                // This check assumes that there's a file
                // present without form data. If form data
                // is present, this method immediately fails
                // and returns the model error.
                if (!MultipartRequestHelper
                    .HasFileContentDisposition(contentDisposition))
                if (!MultipartRequestHelper
                    .HasFileContentDisposition(contentDisposition))
                {
                    ModelState.AddModelError("File", 
                        $"The request couldn't be processed (Error 2).");
                    // Log error

                    return BadRequest(ModelState);
                }

As is mentioned in the comments above the code, it's checking for form data and then when it finds it... it dies. So i've been playing around with the HTML page which looked like this:

<form id="uploadForm" action="Streaming/UploadPhysical" method="post" 
    enctype="multipart/form-data" onsubmit="AJAXSubmit(this);return false;">
    <dl>
        <dt>
            <label for="file">File</label>
        </dt>
        <dd>
            <input id="file" type="file" name="file" />asdfasdf
        </dd>
    </dl>

    <input class="btn" type="submit" value="Upload" />

    <div style="margin-top:15px">
        <output form="uploadForm" name="result"></output>
    </div>
</form>

And I've tried to remove the form like this:

<dl>
    <dt>
        <label for="file">File</label>
    </dt>
    <dd>
        <input id="file" type="file" name="file" />
    </dd>
</dl>

<input class="btn" type="button" asp-controller="Streaming" asp-action="UploadPhysical" value="Upload" />

<div style="margin-top:15px">
    <output form="uploadForm" name="result"></output>
</div>

But the button doesn't do anything now when I click it.

Also, in case you're wondering / it helps, I manually copied in a file into the c:\files folder on my computer and when the sample app opens, it does list the file - proving it can read the folder. I added read /write permissions so hopefully the web app can write to it when I get that far.

Answer 1

I've implemented a similar large file controller but using mongoDB GridFS. It's an amazing software.

Back to your questions:

The entire file is read into an IFormFile, which is a C# representation of the file used to process or save the file.

The resources (disk, memory) used by file uploads depend on the number and size of concurrent file uploads. If an app attempts to buffer too many uploads, the site crashes when it runs out of memory or disk space. If the size or frequency of file uploads is exhausting app resources, use streaming.

source

Example 1:

[HttpPost]
[Authorize]
[DisableRequestSizeLimit]
[RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = int.MaxValue)]
[Route("upload")]
public async Task<ActionResult> UploadFileAsync(IFormFile file)
{  
  if (file == null)
    return Ok(new { success = false, message = "You have to attach a file" });

  var fileName = ContentDispositionHeaderValue.Parse(file.ContentDisposition).FileName.Trim('"');     
  var extension = Path.GetExtension(fileName);
      
  var localPath = $"{Path.Combine(System.AppContext.BaseDirectory, "myCustomDir")}\\{fileName}.{extension}";
  
  // Create if not exists
  Directory.CreateDirectory(Path.Combine(System.AppContext.BaseDirectory, "myCustomDir"));
  
  using (var stream = new FileStream(localPath, FileMode.Create)){
    file.CopyTo(stream);
  }

  //db.SomeContext.Add(someData);
  //await db.SaveChangesAsync();

  return Ok(new { success = true, message = "All set", fileName});      
}  

Example 2 with GridFS:

[HttpPost]
[Authorize]
[DisableRequestSizeLimit]
[RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = int.MaxValue)]
[Route("upload")]
public async Task<ActionResult> UploadFileAsync(IFormFile file)
{
  if (file == null)
    return Ok(new { success = false, message = "You have to attach a file" });

  var options = new GridFSUploadOptions
  {
    Metadata = new BsonDocument("contentType", file.ContentType)
  };

  using (var reader = new StreamReader(file.OpenReadStream()))
  {
    var stream = reader.BaseStream;
    var bsonId = await mongo.GridFs.UploadFromStreamAsync(file.FileName, stream, options);    
  }

  return Ok(new { success = true, message = "All set"});
}

Answer 2

You are on the right path, but as others have pointed out Microsoft have put up a well written document on file uploading which is a must read in your situation - https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0#upload-large-files-with-streaming .

As for your questions

• do you need services.Configure<FormOptions>(x =>

  No you don't! And you don't need services.Configure<IISServerOptions>(options => either, its read from the maxAllowedContentLength that you have configured in your web.config

• When the user picks the file... what is actually happening under the hood? Has the file actually been stuffed into my form and is accessible from my controller?, Is it a stream?

  If you disable the form value model binding and use the MultipartReader the file is streamed and won't be cached into memory or disk, as you drain the stream, more data will be accepted from the client(the browser)

• how do i get to the file?

  Check the document above, there is a working sample for accessing the stream.

• If ultimately, I need to send this file to Sharepoint using this type of an approach (the last example on chunking), it seems that the best approach is to save the file on my server somewhere... and then copy the sample code and try to chunk it out? The sample code seems to be referring to file paths and file sizes, I'm assuming I need to persist it to my web server somewhere first, and then take it from there.

  Not necessarily, using the streaming approach you can copy the stream data directly.