How to upload big files to Azure Blob Storage (.NET Core)

Question

I try to upload 2GB file to Azure Storage, but my code fails and I have an exception. For smaller files it works fine. How to change timeouts and file size limits? I use Azure.Storage.Blobs version 12.10.0 on .NET Core.

My code:

    public FileRepository(AppSettings appSettings)
    {
        var blobServiceClient = new BlobServiceClient(appSettings.StorageConnectionString);

        var containers = blobServiceClient.GetBlobContainers();
        if (!containers.Any(x => x.Name == containerName))
        {
            blobServiceClient.CreateBlobContainer(containerName);
        }
        _blobContainerClient = blobServiceClient.GetBlobContainerClient(containerName);
    }

    public async Task<string> Upload(string fileName, Stream content)
    {
        var blobName = $"{Guid.NewGuid()}/{fileName}";
        await _blobContainerClient.UploadBlobAsync(blobName, content);
        return blobName;
    }

Exception I have:

System.AggregateException
  HResult=0x80131500
  Message=Retry failed after 6 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)
  Source=Azure.Core
  StackTrace:
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.BlobRestClient.BlockBlob.<UploadAsync>d__0.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<UploadInternal>d__26.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass48_0.<<GetPartitionedUploaderBehaviors>b__0>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.PartitionedUploader`2.<UploadInternal>d__19.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.BlobClient.<StagedUploadInternal>d__29.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.BlobContainerClient.<UploadBlobAsync>d__83.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at C:\...\FileRepository.<Upload>d__3.MoveNext() in C:\...\FileRepository.cs:line 38

Inner Exception 1:
TaskCanceledException: The operation was canceled.

Inner Exception 2:
HttpRequestException: Error while copying content to a stream.

Inner Exception 3:
IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..

Inner Exception 4:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

Answer 1

I think it is not possible to control the timeout option and file size limit when uploading using BlobContainerClient .

You will need to make use of UploadAsync method in BlobClient class and specify appropriate values in BlobUploadOptions.TransferOptions .

If you want even more fine grained control over the upload process, you will need to make use of upload methods in specific blob clients. For example, if you're uploading block blobs, you can make use of BlockBlobClient.StageBlockAsync and BlockBlobClient.CommitBlockListAsync .

Answer 2

You could just split the file into chunks/blocks on the client side. Then send lots of small chunks a 100 MB.

Splitting the file into smaller chunks , If you are using a Stream like the code will just read a certain number of bytes and process those until complete.

Please refer Implement File Chunking , Upload a file in blocks and Put Blob for more details.