在Web API中读取文件内容时出现System.OutOfMemoryException

Question

I want to send the content of file as memory stream to S3 bucket via Amazon Firehose. below is my attempt which works fine for small files, but I have a file of 1 GB and I am getting {"Exception of type 'System.OutOfMemoryException' was thrown."} .

My code snippet:

[HttpPost]
public async Task<bool> Upload()
{
    try
    {
        var filesReadToProvider = await Request.Content.ReadAsMultipartAsync();
        foreach (var stream in filesReadToProvider.Contents)
        {
            var fileBytes = await stream.ReadAsByteArrayAsync(); // THIS IS WHERE EXCEPTION COMES
            using (MemoryStream memoryStream = new MemoryStream(fileBytes))
            {
                PutRecordRequest putRecord = new PutRecordRequest();
                putRecord.DeliveryStreamName = myStreamName;
                Record record = new Record();
                record.Data = memoryStream;
                putRecord.Record = record;
                await kinesisClient.PutRecordAsync(putRecord);
            }
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        throw;
    }    
    return true;
}

I looked into this link OutOfMemoryExceptoin but I could not comprehend it. Please help me.

Attempt 1:

var filesReadToProvider = await Request.Content.ReadAsMultipartAsync();
foreach (var stream in filesReadToProvider.Contents)
{
    var fileByte = await stream.ReadAsStreamAsync();
    MemoryStream _ms = new MemoryStream();
    fileByte.CopyTo(_ms); // EXCEPTION HERE
    try
    {
        PutRecordRequest putRecord = new PutRecordRequest();
        putRecord.DeliveryStreamName = myStreamName;
        Record record = new Record();
        record.Data = _ms;
        putRecord.Record = record;
        await kinesisClient.PutRecordAsync(putRecord);
    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed to send record to Kinesis. Exception: {0}", ex.Message);
    }
}

Answer 1

[HttpPost]
public async Task<bool> Upload()
{
    try
    {
        using(var requestStream = await Request.Content.ReadAsStreamAsync())
        {
            PutRecordRequest putRecord = new PutRecordRequest();
            putRecord.DeliveryStreamName = myStreamName;
            Record record = new Record();
            record.Data = requestStream ;
            putRecord.Record = record;
            await kinesisClient.PutRecordAsync(putRecord);

        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        throw;
    }

    return true;
}

This will read the data in chunks. Keep everything in the Stream so you don't keep all the bytes around in a huge array.

Answer 2

When reading large files, I use StreamReader's Readline() method. It works on large files as it manages file system caching internally. Can you use this method, instead? Is there a reason why you are implementing the MemoryStream class? You have a comment asking how to inject the data? Did you try using one of MemoryStream's methods???

https://docs.microsoft.com/en-us/dotnet/api/system.io.memorystream?view=netframework-4.7.2

Update:

Not sure if this is helpful since the code is substantially different from what you are using. But, yours isn't working either, so just a suggestion.

http://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient