C＃內存不足異常將文件存儲為DataSet中的字節數組

Question

我遇到了無法重現的“內存不足”異常問題，但是每次運行單元測試的構建服務器都會命中。 在我的機器上運行單元測試不會導致異常。 所做的更改是因為原始代碼在傳入的流中存在大型PDF的奇怪問題。 如果您對原始代碼為什么會遇到大型PDF的問題或新代碼為何會導致“內存不足”異常的想法，請告訴我。

原始代碼：

// stream is a valid Stream and parentKey is a valid int
// Reset the stream position
stream.Position = 0;
int sequenceNumber = 0;
int StreamReadSize = short.MaxValue;
byte[] buffer = new byte[StreamReadSize]; 
MemoryStream outStream = null;
try
{
    long previousStreamPosition = 0;
    long DataBlockSize  = 52428800;
    int read;
    while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
    {
        if (outStream == null)
            outStream = new MemoryStream(new byte[System.Math.Min(stream.Length - previousStreamPosition, DataBlockSize)]);

        previousStreamPosition = stream.Position;
        outStream.Write(buffer, 0, read);
        if (outStream.Position <= (DataBlockSize - StreamReadSize) && stream.Position < stream.Length)
            continue;

        var dataRow = dataSet.Tables["table_name"].NewRow();
        dataRow["parent_key"] = parentKey;
        dataRow["key"] = FuncThatReturnsNextAvailableKey();
        dataRow["sequence_number"] = ++sequenceNumber;
        // Reset the position and Zip up the data
        outStream.Position = 0;

        dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(outStream);

        dataSet.Tables["table_name"].Rows.Add(dataRow);

        outStream.Flush();
        outStream.Dispose();
        outStream = null;
    }
}
finally
{
    if (outStream != null)
        outStream.Dispose();
}

新代碼：

// stream is a valid Stream and parentKey is a valid int
// Reset the stream position and create the variables needed for saving the file data
stream.Position = 0;
int sequenceNumber = 0;
int bytesRead;
int DataBlockSize = 52428800;
byte[] buffer = new byte[DataBlockSize];
while ((bytesRead = stream.Read(buffer, 0, DataBlockSize)) > 0)
{
    sequenceNumber++;

    // Create and initialize the row
    var dataRow = dataSet.Tables["table_name"].NewRow();
    dataRow["parent_key"] = parentKey;
    dataRow["key"] = FuncThatReturnsNextAvailableKey(); ;
    dataRow["sequence_number"] = sequenceNumber;

    // If the stream reads in less data than the size of the buffer then create an appropriately sized version of the buffer
    // that will only hold the data that was read in
    if (bytesRead != DataBlockSize)
    {
        var shrunkBuffer = new byte[bytesRead];
        Array.Copy(buffer, shrunkBuffer, bytesRead);
        using (var memoryStream = new MemoryStream(shrunkBuffer))
            dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
    }
    else
    {
        using (var memoryStream = new MemoryStream(buffer))
            dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
    }

    // Add the finished row
    dataSet.Tables["table_name"].Rows.Add(dataRow);
}

Answer 1

有兩個不同的環境可能會產生不同的結果是有道理的。 可能是您的構建服務器的內存少於您的個人編碼環境。

可能是您通過以下方式將字節數組保留在內存中：

dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);

您正在布置輸出流，但是我假設您的數據行保留在內存中，因此您要保留對該字節數組的引用。 可能是因為多個PDF達到了您的過程可以自行分配的最大分配量。

Answer 2

使用類記憶庫

來自https://gist.github.com/bittercoder/3588074的來源

        using (System.IO.FileStream stream = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read))
        {
            using (MemoryTributary memT = new MemoryTributary())
            {

                memT.ReadFrom(stream, stream.Length);
                return memT.ToArray();
            }
        }