Efficiently read all available bytes from binary stream in python (like C++ readsome)

Question

I have a character device which usually delivers multiple megabytes per second of binary data. However, the device may report an error with just a few bytes. Additionally, the stream may stall for an indefinite amount of time – this is not an error. The binary data format is proprietary and I cannot parse it. My glue application is written in Python 3 and I need to relay the data.

Currently, I use the select module to check for available data before reading:

keep_running = True
while keep_running:
    chunk = b''
    if (input_stream in select.select([input_stream], [], [], 1)[0]):
        chunk = input_stream.read(BUFFER_SIZE)

This works fine for normal operation and for a stalled stream. However, if I chose a buffer size of greater than one byte, the read will block potentially forever if the number of available bytes is greater than 0 but less than the buffer size.

Blocking for a finite amount of time is okay, but I need to have some way of aborting the loop gracefully ( without closing the stream from another thread). As far as I know, a blocking read() cannot be aborted.

My gut says, a buffer size of 1 is detrimental to performance. Is there a better way to do this? Essentially, I am looking for a Python equivalent of C++'s readsome .

Maybe I am tackling this from the wrong angle. Alternative approaches are welcome, too.

Answer 1

I was just reading this article .

Try using the iter() function with its second argument.

for chunk in iter(lambda: input_stream.read(BUFFER_SIZE), b''):
    dostuff(chunk)