Python-如何在io.BufferedReader中使用自定義buffer_size？

Question

據我了解， io.BufferedReader的buffer_size參數應該控制傳遞給基礎讀取器的讀取緩沖區大小。

但是，我沒有看到這種行為。 相反，當我reader.read()整個文件時，將使用io.DEFAULT_BUFFER_SIZE並忽略buffer_size 。 當我reader.read(length) ， length用作緩沖區大小，並且buffer_size參數再次被忽略。

最小示例：

import io

class MyReader(io.RawIOBase):

    def __init__(self, length):
        self.length = length
        self.position = 0

    def readinto(self, b):
        print('read buffer length: %d' % len(b))
        length = min(len(b), self.length - self.position)
        self.position += length
        b[:length] = 'a' * length
        return length

    def readable(self):
        return True

    def seekable(self):
        return False


print('# read entire file')
reader = io.BufferedReader(MyReader(20000), buffer_size=100)
print('output length: %d' % len(reader.read()))

print('\n# read part of file file')
reader = io.BufferedReader(MyReader(20000), buffer_size=100)
print('output length: %d' % len(reader.read(10000)))

print('\n# read beyond end of file file')
reader = io.BufferedReader(MyReader(20000), buffer_size=100)
print 'output length: %d' % len(reader.read(30000))

輸出：

# read entire file
read buffer length: 8192
read buffer length: 8192
read buffer length: 8192
read buffer length: 8192
read buffer length: 8192
output length: 20000

# read part of file file
read buffer length: 10000
output length: 10000

# read beyond end of file file
read buffer length: 30000
read buffer length: 10000
output length: 20000

我是否誤解了BufferedReader應該如何工作？

Answer 1

BufferedIOReader是保留一個內部緩沖區，然后設置該緩沖區的大小。 該緩沖區用於滿足較小的讀取，以避免在速度較慢的I / O設備上進行許多讀取調用。

但是，緩沖區不會嘗試限制讀取的大小！

從io.BufferedIOReader文檔中 ：

從此對象讀取數據時，可能會從底層原始流中請求大量數據，並將其保存在內部緩沖區中。 然后可以在后續讀取時直接返回緩沖的數據。

該對象繼承自io.BufferedIOBase ，狀態為：

RawIOBase的主要區別在於，方法read() ， readinto()和write()將嘗試（分別）讀取請求的盡可能多的輸入或消耗所有給定的輸出，但以進行多個系統調用為代價。

因為您在對象上調用了.read() ，所以從包裝的對象中讀取了更大的塊，以讀取所有數據到最后。 BufferedIOReader()實例保存的內部緩沖區在這里不起作用，您畢竟需要所有數據。

如果讀取較小的塊，緩沖區將起作用：

>>> reader = io.BufferedReader(MyReader(2048), buffer_size=512)
>>> __ = reader.read(42)  # initial read, fill buffer
read buffer length: 512
>>> __ = reader.read(123)  # within the buffer, no read to underlying file needed
>>> __ = reader.read(456)  # deplete buffer, another read needed to re-fill
read buffer length: 512
>>> __ = reader.read(123)  # within the buffer, no read to underlying file needed
>>> __ = reader.read()     # read until end, uses larger blocks to read from wrapped file
read buffer length: 8192
read buffer length: 8192
read buffer length: 8192