“Chunked”MemoryStream

Question

我正在尋找MemoryStream的實現，它不會將內存分配為一個大塊，而是一個塊的集合。 我想在內存（64位）中存儲幾GB的數據，並避免內存碎片的限制。

Answer 1

像這樣的東西：

class ChunkedMemoryStream : Stream
{
    private readonly List<byte[]> _chunks = new List<byte[]>();
    private int _positionChunk;
    private int _positionOffset;
    private long _position;

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    public override void Flush() { }

    public override long Length
    {
        get { return _chunks.Sum(c => c.Length); }
    }

    public override long Position
    {
        get
        {
            return _position;
        }
        set
        {
            _position = value;

            _positionChunk = 0;

            while (_positionOffset != 0)
            {
                if (_positionChunk >= _chunks.Count)
                    throw new OverflowException();

                if (_positionOffset < _chunks[_positionChunk].Length)
                    return;

                _positionOffset -= _chunks[_positionChunk].Length;
                _positionChunk++;
            }
        }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int result = 0;
        while ((count != 0) && (_positionChunk != _chunks.Count))
        {
            int fromChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
            if (fromChunk != 0)
            {
                Array.Copy(_chunks[_positionChunk], _positionOffset, buffer, offset, fromChunk);
                offset += fromChunk;
                count -= fromChunk;
                result += fromChunk;
                _position += fromChunk;
            }

            _positionOffset = 0;
            _positionChunk++;
        }
        return result;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        long newPos = 0;

        switch (origin)
        {
            case SeekOrigin.Begin:
                newPos = offset;
                break;
            case SeekOrigin.Current:
                newPos = Position + offset;
                break;
            case SeekOrigin.End:
                newPos = Length - offset;
                break;
        }

        Position = Math.Max(0, Math.Min(newPos, Length));
        return newPos;
    }

    public override void SetLength(long value)
    {
        throw new NotImplementedException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        while ((count != 0) && (_positionChunk != _chunks.Count))
        {
            int toChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
            if (toChunk != 0)
            {
                Array.Copy(buffer, offset, _chunks[_positionChunk], _positionOffset, toChunk);
                offset += toChunk;
                count -= toChunk;
                _position += toChunk;
            }

            _positionOffset = 0;
            _positionChunk++;
        }

        if (count != 0)
        {
            byte[] chunk = new byte[count];
            Array.Copy(buffer, offset, chunk, 0, count);
            _chunks.Add(chunk);
            _positionChunk = _chunks.Count;
            _position += count;
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        ChunkedMemoryStream cms = new ChunkedMemoryStream();

        Debug.Assert(cms.Length == 0);
        Debug.Assert(cms.Position == 0);

        cms.Position = 0;

        byte[] helloworld = Encoding.UTF8.GetBytes("hello world");

        cms.Write(helloworld, 0, 3);
        cms.Write(helloworld, 3, 3);
        cms.Write(helloworld, 6, 5);

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 11);

        cms.Position = 0;

        byte[] b = new byte[20];
        cms.Read(b, 3, (int)cms.Length);
        Debug.Assert(b.Skip(3).Take(11).SequenceEqual(helloworld));

        cms.Position = 0;
        cms.Write(Encoding.UTF8.GetBytes("seeya"), 0, 5);

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 5);

        cms.Position = 0;
        cms.Read(b, 0, (byte) cms.Length);
        Debug.Assert(b.Take(11).SequenceEqual(Encoding.UTF8.GetBytes("seeya world")));

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 11);

        cms.Write(Encoding.UTF8.GetBytes(" again"), 0, 6);

        Debug.Assert(cms.Length == 17);
        Debug.Assert(cms.Position == 17);

        cms.Position = 0;
        cms.Read(b, 0, (byte)cms.Length);
        Debug.Assert(b.Take(17).SequenceEqual(Encoding.UTF8.GetBytes("seeya world again")));

    }
}

Answer 2

您需要首先確定虛擬地址碎片是否是問題。

如果你是64位機器（你似乎表明你是），我非常懷疑它是什么。 每個64位進程幾乎都有可用的整個64位虛擬內存空間，您唯一擔心的是虛擬地址空間碎片而不是物理內存碎片（這是操作系統必須擔心的）。 操作系統內存管理器已經將內存分頁。 對於可預見的未來，在物理內存耗盡之前，您不會耗盡虛擬地址空間。 在我們退休之前，這不太可能發生變化。

如果您有32位地址空間，然后在GB ramge中分配連續的大塊內存，您將很快遇到碎片問題。 在CLR中沒有庫存塊分配內存流。 ASP.NET中有一個（由於其他原因）但它無法訪問。 如果你必須走這條路，你可能最好自己寫一個，因為你的應用程序的使用模式不太可能與許多其他類似，並且試圖將你的數據放入32位地址空間可能是你的性能瓶頸。

如果您正在操作GB數據，我強烈建議您需要64位進程。 無論您是多么敏銳，它都會比手動解決方案更好地完成32位地址空間碎片。

Answer 3

Bing團隊發布了RecyclableMemoryStream並在此處寫了相關內容。 他們引用的好處是：

1. 使用池化緩沖區消除大對象堆分配
2. 導致2代GC的數量要少得多，並且因GC而停頓的時間要少得多
3. 通過限制池大小避免內存泄漏
4. 避免內存碎片
5. 提供出色的可調試性
6. 提供績效跟蹤指標

Answer 4

我在我的應用程序中發現了類似的問題。 我已經閱讀了大量的壓縮數據，並且使用MemoryStream遭遇了OutOfMemoryException。 我根據字節數組的集合編寫了自己的“chunked”內存流實現。 如果你有任何想法如何使這個內存流更有效，請寫信告訴我。

    public sealed class ChunkedMemoryStream : Stream
{
    #region Constants

    private const int BUFFER_LENGTH = 65536;
    private const byte ONE = 1;
    private const byte ZERO = 0;

    #endregion

    #region Readonly & Static Fields

    private readonly Collection<byte[]> _chunks;

    #endregion

    #region Fields

    private long _length;

    private long _position;
    private const byte TWO = 2;

    #endregion

    #region C'tors

    public ChunkedMemoryStream()
    {
        _chunks = new Collection<byte[]> { new byte[BUFFER_LENGTH], new byte[BUFFER_LENGTH] };
        _position = ZERO;
        _length = ZERO;
    }

    #endregion

    #region Instance Properties

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    public override long Length
    {
        get { return _length; }
    }

    public override long Position
    {
        get { return _position; }
        set
        {
            if (!CanSeek)
                throw new NotSupportedException();

            _position = value;

            if (_position > _length)
                _position = _length - ONE;
        }
    }


    private byte[] CurrentChunk
    {
        get
        {
            long positionDividedByBufferLength = _position / BUFFER_LENGTH;
            var chunkIndex = Convert.ToInt32(positionDividedByBufferLength);
            byte[] chunk = _chunks[chunkIndex];
            return chunk;
        }
    }

    private int PositionInChunk
    {
        get
        {
            int positionInChunk = Convert.ToInt32(_position % BUFFER_LENGTH);
            return positionInChunk;
        }
    }

    private int RemainingBytesInCurrentChunk
    {
        get
        {
            Contract.Ensures(Contract.Result<int>() > ZERO);
            int remainingBytesInCurrentChunk = CurrentChunk.Length - PositionInChunk;
            return remainingBytesInCurrentChunk;
        }
    }

    #endregion

    #region Instance Methods

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (offset + count > buffer.Length)
            throw new ArgumentException();

        if (buffer == null)
            throw new ArgumentNullException();

        if (offset < ZERO || count < ZERO)
            throw new ArgumentOutOfRangeException();

        if (!CanRead)
            throw new NotSupportedException();

        int bytesToRead = count;
        if (_length - _position < bytesToRead)
            bytesToRead = Convert.ToInt32(_length - _position);

        int bytesreaded = 0;
        while (bytesToRead > ZERO)
        {
            // get remaining bytes in current chunk
            // read bytes in current chunk
            // advance to next position
            int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;
            if (remainingBytesInCurrentChunk > bytesToRead)
                remainingBytesInCurrentChunk = bytesToRead;
            Array.Copy(CurrentChunk, PositionInChunk, buffer, offset, remainingBytesInCurrentChunk);
            //move position in source
            _position += remainingBytesInCurrentChunk;
            //move position in target
            offset += remainingBytesInCurrentChunk;
            //bytesToRead is smaller
            bytesToRead -= remainingBytesInCurrentChunk;
            //count readed bytes;
            bytesreaded += remainingBytesInCurrentChunk;
        }
        return bytesreaded;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        switch (origin)
        {
            case SeekOrigin.Begin:
                Position = offset;
                break;
            case SeekOrigin.Current:
                Position += offset;
                break;
            case SeekOrigin.End:
                Position = Length + offset;
                break;
        }
        return Position;
    }

    private long Capacity
    {
        get
        {
            int numberOfChunks = _chunks.Count;


            long capacity = numberOfChunks * BUFFER_LENGTH;
            return capacity;
        }
    }

    public override void SetLength(long value)
    {
        if (value > _length)
        {
            while (value > Capacity)
            {
                var item = new byte[BUFFER_LENGTH];
                _chunks.Add(item);
            }
        }
        else if (value < _length)
        {
            var decimalValue = Convert.ToDecimal(value);
            var valueToBeCompared = decimalValue % BUFFER_LENGTH == ZERO ? Capacity : Capacity - BUFFER_LENGTH;
            //remove data chunks, but leave at least two chunks
            while (value < valueToBeCompared && _chunks.Count > TWO)
            {
                byte[] lastChunk = _chunks.Last();
                _chunks.Remove(lastChunk);
            }
        }
        _length = value;
        if (_position > _length - ONE)
            _position = _length == 0 ? ZERO : _length - ONE;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        if (!CanWrite)
            throw new NotSupportedException();

        int bytesToWrite = count;

        while (bytesToWrite > ZERO)
        {
            //get remaining space in current chunk
            int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;

            //if count of bytes to be written is fewer than remaining
            if (remainingBytesInCurrentChunk > bytesToWrite)
                remainingBytesInCurrentChunk = bytesToWrite;

            //if remaining bytes is still greater than zero
            if (remainingBytesInCurrentChunk > ZERO)
            {
                //write remaining bytes to current Chunk

                Array.Copy(buffer, offset, CurrentChunk, PositionInChunk, remainingBytesInCurrentChunk);

                //change offset of source array
                offset += remainingBytesInCurrentChunk;
                //change bytes to write
                bytesToWrite -= remainingBytesInCurrentChunk;
                //change length and position
                _length += remainingBytesInCurrentChunk;
                _position += remainingBytesInCurrentChunk;
            }

            if (Capacity == _position)
                _chunks.Add(new byte[BUFFER_LENGTH]);
        }
    }

    /// <summary>
    ///     Gets entire content of stream regardless of Position value and return output as byte array
    /// </summary>
    /// <returns>byte array</returns>
    public byte[] ToArray()
    {
        var outputArray = new byte[Length];
        if (outputArray.Length != ZERO)
        {
            long outputPosition = ZERO;
            foreach (byte[] chunk in _chunks)
            {
                var remainingLength = (Length - outputPosition) > chunk.Length
                                          ? chunk.Length
                                          : Length - outputPosition;
                Array.Copy(chunk, ZERO, outputArray, outputPosition, remainingLength);
                outputPosition = outputPosition + remainingLength;
            }
        }
        return outputArray;
    }

    /// <summary>
    ///     Method set Position to first element and write entire stream to another
    /// </summary>
    /// <param name="stream">Target stream</param>
    public void WriteTo(Stream stream)
    {
        Contract.Requires(stream != null);

        Position = ZERO;
        var buffer = new byte[BUFFER_LENGTH];
        int bytesReaded;
        do
        {
            bytesReaded = Read(buffer, ZERO, BUFFER_LENGTH);
            stream.Write(buffer, ZERO, bytesReaded);
        } while (bytesReaded > ZERO);
    }

    #endregion
}

Answer 5

這是一個完整的實現：

/// <summary>
/// Defines a MemoryStream that does not sit on the Large Object Heap, thus avoiding memory fragmentation.
/// </summary>
public sealed class ChunkedMemoryStream : Stream
{
    /// <summary>
    /// Defines the default chunk size. Currently defined as 0x10000.
    /// </summary>
    public const int DefaultChunkSize = 0x10000; // needs to be < 85000

    private List<byte[]> _chunks = new List<byte[]>();
    private long _position;
    private int _chunkSize;
    private int _lastChunkPos;
    private int _lastChunkPosIndex;

    /// <summary>
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class.
    /// </summary>
    public ChunkedMemoryStream()
        : this(DefaultChunkSize)
    {
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class.
    /// </summary>
    /// <param name="chunkSize">Size of the underlying chunks.</param>
    public ChunkedMemoryStream(int chunkSize)
        : this(null)
    {
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array.
    /// </summary>
    /// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param>
    public ChunkedMemoryStream(byte[] buffer)
        : this(DefaultChunkSize, buffer)
    {
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array.
    /// </summary>
    /// <param name="chunkSize">Size of the underlying chunks.</param>
    /// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param>
    public ChunkedMemoryStream(int chunkSize, byte[] buffer)
    {
        FreeOnDispose = true;
        ChunkSize = chunkSize;
        _chunks.Add(new byte[chunkSize]);
        if (buffer != null)
        {
            Write(buffer, 0, buffer.Length);
            Position = 0;
        }
    }

    /// <summary>
    /// Gets or sets a value indicating whether to free the underlying chunks on dispose.
    /// </summary>
    /// <value><c>true</c> if [free on dispose]; otherwise, <c>false</c>.</value>
    public bool FreeOnDispose { get; set; }

    /// <summary>
    /// Releases the unmanaged resources used by the <see cref="T:System.IO.Stream"/> and optionally releases the managed resources.
    /// </summary>
    /// <param name="disposing">true to release both managed and unmanaged resources; false to release only unmanaged resources.</param>
    protected override void Dispose(bool disposing)
    {
        if (FreeOnDispose)
        {
            if (_chunks != null)
            {
                _chunks = null;
                _chunkSize = 0;
                _position = 0;
            }
        }
        base.Dispose(disposing);
    }

    /// <summary>
    /// When overridden in a derived class, clears all buffers for this stream and causes any buffered data to be written to the underlying device.
    /// This implementation does nothing.
    /// </summary>
    public override void Flush()
    {
        // do nothing
    }

    /// <summary>
    /// When overridden in a derived class, reads a sequence of bytes from the current stream and advances the position within the stream by the number of bytes read.
    /// </summary>
    /// <param name="buffer">An array of bytes. When this method returns, the buffer contains the specified byte array with the values between <paramref name="offset"/> and (<paramref name="offset"/> + <paramref name="count"/> - 1) replaced by the bytes read from the current source.</param>
    /// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin storing the data read from the current stream.</param>
    /// <param name="count">The maximum number of bytes to be read from the current stream.</param>
    /// <returns>
    /// The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
    /// </returns>
    /// <exception cref="T:System.ArgumentException">
    /// The sum of <paramref name="offset"/> and <paramref name="count"/> is larger than the buffer length.
    /// </exception>
    /// <exception cref="T:System.ArgumentNullException">
    ///     <paramref name="buffer"/> is null.
    /// </exception>
    /// <exception cref="T:System.ArgumentOutOfRangeException">
    ///     <paramref name="offset"/> or <paramref name="count"/> is negative.
    /// </exception>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override int Read(byte[] buffer, int offset, int count)
    {
        if (buffer == null)
            throw new ArgumentNullException("buffer");

        if (offset < 0)
            throw new ArgumentOutOfRangeException("offset");

        if (count < 0)
            throw new ArgumentOutOfRangeException("count");

        if ((buffer.Length - offset) < count)
            throw new ArgumentException(null, "count");

        CheckDisposed();

        int chunkIndex = (int)(_position / ChunkSize);
        if (chunkIndex == _chunks.Count)
            return 0;

        int chunkPos = (int)(_position % ChunkSize);
        count = (int)Math.Min(count, Length - _position);
        if (count == 0)
            return 0;

        int left = count;
        int inOffset = offset;
        int total = 0;
        do
        {
            int toCopy = Math.Min(left, ChunkSize - chunkPos);
            Buffer.BlockCopy(_chunks[chunkIndex], chunkPos, buffer, inOffset, toCopy);
            inOffset += toCopy;
            left -= toCopy;
            total += toCopy;
            if ((chunkPos + toCopy) == ChunkSize)
            {
                if (chunkIndex == (_chunks.Count - 1))
                {
                    // last chunk
                    break;
                }
                chunkPos = 0;
                chunkIndex++;
            }
            else
            {
                chunkPos += toCopy;
            }
        }
        while (left > 0);
        _position += total;
        return total;
    }

    /// <summary>
    /// Reads a byte from the stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream.
    /// </summary>
    /// <returns>
    /// The unsigned byte cast to an Int32, or -1 if at the end of the stream.
    /// </returns>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override int ReadByte()
    {
        CheckDisposed();
        if (_position >= Length)
            return -1;

        byte b = _chunks[(int)(_position / ChunkSize)][_position % ChunkSize];
        _position++;
        return b;
    }

    /// <summary>
    /// When overridden in a derived class, sets the position within the current stream.
    /// </summary>
    /// <param name="offset">A byte offset relative to the <paramref name="origin"/> parameter.</param>
    /// <param name="origin">A value of type <see cref="T:System.IO.SeekOrigin"/> indicating the reference point used to obtain the new position.</param>
    /// <returns>
    /// The new position within the current stream.
    /// </returns>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override long Seek(long offset, SeekOrigin origin)
    {
        CheckDisposed();
        switch (origin)
        {
            case SeekOrigin.Begin:
                Position = offset;
                break;

            case SeekOrigin.Current:
                Position += offset;
                break;

            case SeekOrigin.End:
                Position = Length + offset;
                break;
        }
        return Position;
    }

    private void CheckDisposed()
    {
        if (_chunks == null)
            throw new ObjectDisposedException(null, "Cannot access a disposed stream");
    }

    /// <summary>
    /// When overridden in a derived class, sets the length of the current stream.
    /// </summary>
    /// <param name="value">The desired length of the current stream in bytes.</param>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override void SetLength(long value)
    {
        CheckDisposed();
        if (value < 0)
            throw new ArgumentOutOfRangeException("value");

        if (value > Length)
            throw new ArgumentOutOfRangeException("value");

        long needed = value / ChunkSize;
        if ((value % ChunkSize) != 0)
        {
            needed++;
        }

        if (needed > int.MaxValue)
            throw new ArgumentOutOfRangeException("value");

        if (needed < _chunks.Count)
        {
            int remove = (int)(_chunks.Count - needed);
            for (int i = 0; i < remove; i++)
            {
                _chunks.RemoveAt(_chunks.Count - 1);
            }
        }
        _lastChunkPos = (int)(value % ChunkSize);
    }

    /// <summary>
    /// Converts the current stream to a byte array.
    /// </summary>
    /// <returns>An array of bytes</returns>
    public byte[] ToArray()
    {
        CheckDisposed();
        byte[] bytes = new byte[Length];
        int offset = 0;
        for (int i = 0; i < _chunks.Count; i++)
        {
            int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length;
            if (count > 0)
            {
                Buffer.BlockCopy(_chunks[i], 0, bytes, offset, count);
                offset += count;
            }
        }
        return bytes;
    }

    /// <summary>
    /// When overridden in a derived class, writes a sequence of bytes to the current stream and advances the current position within this stream by the number of bytes written.
    /// </summary>
    /// <param name="buffer">An array of bytes. This method copies <paramref name="count"/> bytes from <paramref name="buffer"/> to the current stream.</param>
    /// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin copying bytes to the current stream.</param>
    /// <param name="count">The number of bytes to be written to the current stream.</param>
    /// <exception cref="T:System.ArgumentException">
    /// The sum of <paramref name="offset"/> and <paramref name="count"/> is greater than the buffer length.
    /// </exception>
    /// <exception cref="T:System.ArgumentNullException">
    ///     <paramref name="buffer"/> is null.
    /// </exception>
    /// <exception cref="T:System.ArgumentOutOfRangeException">
    ///     <paramref name="offset"/> or <paramref name="count"/> is negative.
    /// </exception>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override void Write(byte[] buffer, int offset, int count)
    {
        if (buffer == null)
            throw new ArgumentNullException("buffer");

        if (offset < 0)
            throw new ArgumentOutOfRangeException("offset");

        if (count < 0)
            throw new ArgumentOutOfRangeException("count");

        if ((buffer.Length - offset) < count)
            throw new ArgumentException(null, "count");

        CheckDisposed();

        int chunkPos = (int)(_position % ChunkSize);
        int chunkIndex = (int)(_position / ChunkSize);
        if (chunkIndex == _chunks.Count)
        {
            _chunks.Add(new byte[ChunkSize]);
        }

        int left = count;
        int inOffset = offset;
        do
        {
            int copied = Math.Min(left, ChunkSize - chunkPos);
            Buffer.BlockCopy(buffer, inOffset, _chunks[chunkIndex], chunkPos, copied);
            inOffset += copied;
            left -= copied;
            if ((chunkPos + copied) == ChunkSize)
            {
                chunkIndex++;
                chunkPos = 0;
                if (chunkIndex == _chunks.Count)
                {
                    _chunks.Add(new byte[ChunkSize]);
                }
            }
            else
            {
                chunkPos += copied;
            }
        }
        while (left > 0);
        _position += count;
        if (chunkIndex == (_chunks.Count - 1))
        {
            if ((chunkIndex > _lastChunkPosIndex) ||
                ((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos)))
            {
                _lastChunkPos = chunkPos;
                _lastChunkPosIndex = chunkIndex;
            }
        }
    }

    /// <summary>
    /// Writes a byte to the current position in the stream and advances the position within the stream by one byte.
    /// </summary>
    /// <param name="value">The byte to write to the stream.</param>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override void WriteByte(byte value)
    {
        CheckDisposed();
        int chunkIndex = (int)(_position / ChunkSize);
        int chunkPos = (int)(_position % ChunkSize);

        if (chunkPos > (ChunkSize - 1)) //changed from (chunkPos >= (ChunkSize - 1))
        {
            chunkIndex++;
            chunkPos = 0;
            if (chunkIndex == _chunks.Count)
            {
                _chunks.Add(new byte[ChunkSize]);
            }
        }
        _chunks[chunkIndex][chunkPos++] = value;
        _position++;
        if (chunkIndex == (_chunks.Count - 1))
        {
            if ((chunkIndex > _lastChunkPosIndex) ||
                ((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos)))
            {
                _lastChunkPos = chunkPos;
                _lastChunkPosIndex = chunkIndex;
            }
        }
    }

    /// <summary>
    /// Writes to the specified stream.
    /// </summary>
    /// <param name="stream">The stream.</param>
    public void WriteTo(Stream stream)
    {
        if (stream == null)
            throw new ArgumentNullException("stream");

        CheckDisposed();
        for (int i = 0; i < _chunks.Count; i++)
        {
            int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length;
            stream.Write(_chunks[i], 0, count);
        }
    }

    /// <summary>
    /// When overridden in a derived class, gets a value indicating whether the current stream supports reading.
    /// </summary>
    /// <value></value>
    /// <returns>true if the stream supports reading; otherwise, false.
    /// </returns>
    public override bool CanRead
    {
        get
        {
            return true;
        }
    }

    /// <summary>
    /// When overridden in a derived class, gets a value indicating whether the current stream supports seeking.
    /// </summary>
    /// <value></value>
    /// <returns>true if the stream supports seeking; otherwise, false.
    /// </returns>
    public override bool CanSeek
    {
        get
        {
            return true;
        }
    }

    /// <summary>
    /// When overridden in a derived class, gets a value indicating whether the current stream supports writing.
    /// </summary>
    /// <value></value>
    /// <returns>true if the stream supports writing; otherwise, false.
    /// </returns>
    public override bool CanWrite
    {
        get
        {
            return true;
        }
    }

    /// <summary>
    /// When overridden in a derived class, gets the length in bytes of the stream.
    /// </summary>
    /// <value></value>
    /// <returns>
    /// A long value representing the length of the stream in bytes.
    /// </returns>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override long Length
    {
        get
        {
            CheckDisposed();
            if (_chunks.Count == 0)
                return 0;

            return (_chunks.Count - 1) * ChunkSize + _lastChunkPos;
        }
    }

    /// <summary>
    /// Gets or sets the size of the underlying chunks. Cannot be greater than or equal to 85000.
    /// </summary>
    /// <value>The chunks size.</value>
    public int ChunkSize
    {
        get
        {
            return _chunkSize;
        }
        set
        {
            if ((value <= 0) || (value >= 85000))
                throw new ArgumentOutOfRangeException("value");

            _chunkSize = value;
        }
    }

    /// <summary>
    /// When overridden in a derived class, gets or sets the position within the current stream.
    /// </summary>
    /// <value></value>
    /// <returns>
    /// The current position within the stream.
    /// </returns>
    /// <exception cref="T:System.ObjectDisposedException">
    /// Methods were called after the stream was closed.
    /// </exception>
    public override long Position
    {
        get
        {
            CheckDisposed();
            return _position;
        }
        set
        {
            CheckDisposed();
            if (value < 0)
                throw new ArgumentOutOfRangeException("value");

            if (value > Length)
                throw new ArgumentOutOfRangeException("value");

            _position = value;
        }
    }
}

Answer 6

你可以用它

http://www.123aspx.com/Rotor/src/chunkedmemorystream.cs.txt

Answer 7

在處理超過2GB的內存塊時應該使用UnmanagedMemoryStream ，因為MemoryStream限制為2GB，並且UnmanagedMemoryStream用於處理此問題。

Answer 8

SparseMemoryStream在.NET中執行此操作，但它深埋在內部類庫中 - 當然，源代碼是可用的，因為Microsoft將其全部作為開源來實現。

你可以在這里獲取代碼： http ： //www.dotnetframework.org/default.aspx/4@0/4@0/DEVDIV_TFS/Dev10/Releases/RTMRel/wpf/src/Base/MS/Internal/IO /包裝/ SparseMemoryStream @ CS / 1305600 / SparseMemoryStream @ CS

話雖這么說，我強烈建議不要按原樣使用它 - 至少刪除所有對啟動器的IsolatedStorage調用，因為這似乎是框架的打包API中無法結束錯誤*的原因。

（*：除了在流中傳播數據外，如果它變得太大，它基本上會出於某種原因重新發明交換文件 - 在用戶的隔離存儲中不會少 - 而巧合的是，大多數允許基於.NET的MS產品加載項沒有以可以訪問獨立存儲的方式設置其應用程序域 - 例如，VSTO加載項因此問題而臭名昭着。）

Answer 9

分塊流的另一個實現可以被視為庫存MemoryStream替換。 此外，它允許在LOH上分配一個大字節數組，它將用作“chunk”池，在所有ChunkedStream實例之間共享...

https://github.com/ImmortalGAD/ChunkedStream