Azure Function - System.OutOfMemoryException

Question

I have an Az Function that I'm having problems processing large size files.

The snippet where I download the file is this:

    public byte[] DownloadBlobBytes(BlobContainerClient blobContainerClient, string blobName)
    {
        BlobClient blobClient = blobContainerClient.GetBlobClient(blobName);

        if (!blobClient.Exists())
        {
            throw new FileNotFoundException("File not found in Azure Blob Storage.", blobName);
        }

        _logger.LogDebug("Starting download of blob '{BlobUri}' from Azure Blob Storage.", blobClient.Uri);

        return blobClient.DownloadContent().Value.Content.ToArray();
    }

It throws this error when running in production environment:

    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at System.BinaryData.FromStreamAsync(Stream stream, Boolean async, CancellationToken cancellationToken)
   at System.BinaryData.FromStream(Stream stream)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContentInternal(BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContent(BlobRequestConditions conditions, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContent(CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContent()
   at ThomsonReuters.DocMi.Infrastructure.Communication.Azure.AzureBlobStorageClient.DownloadBlobBytes(BlobContainerClient blobContainerClient, String blobName)

Locally it works perfectly but in production it doesn't.

Searching I saw that in the documentation Azure Functions limits there is a memory limit of 1.5GB per instance.

Can I solve this problem that only happens in production?

Is there anything I can do to reduce memory usage when downloading the bytes from the file?

Answer 1

1.5GB is the memory available on a consumption plan. If you deploy the function to an app service plan/dedicated plan, you can get more. But there may be cost implications there so you'd need to consider that.

With this kind of situation it usually boils down to 2 core options:

1. splitting large files into smaller chunks, and then processing those to reduce resources needed

2. work with streams - if you have a 2GB file, the approach you're using will never work as it's trying to load the whole file into memory. Instead, use blobClient.OpenRead ( docs ) - this will only download contents as you read from the stream.

For example, if you're trying to download a file from 1 source and store it somewhere else, you should be able to use the stream to stream the contents from 1 to the other without downloading all into memory. Or if you're trying to do something like do some processing on the file - use the stream to read the contents in chunks (eg read 100 lines, do some processing, read the next 100)