Non-blocking I/O with timeout

Question

I'm trying to read a pipe in non-blocking mode. Here is a similar question and answer but it uses threads Non-blocking read on a subprocess.PIPE in python

I tried the following and looks simpler than using threads but is non-blocking only if the output is line buffered - not sure if I'm doing this wrong so can someone point me in the right direction?

#!/usr/bin/python

import select
from subprocess import *
import time

# do non-blocking read but timeout after some time as we don't want to poll forever   
timeout = 4
READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR

poller = select.poll()
proc = Popen("./output.sh".split(), stdout=PIPE);
poller.register(proc.stdout, READ_ONLY)

now = time.time()
end = now + timeout

while time.time() < end:
    if poller.poll(timeout):
        # works as expected as long as output.sh produces lines
        # read() also blocks
        print "%s" % proc.stdout.readline(),

proc.kill()

output.sh is what generates output

#!/bin/bash

for i in `seq 1 400`;
do
    sleep 1;
    # doesn't have newlines
    echo -n $i
done

Answer 1

The poll() function indicates you have at least one byte ready to read. If you call readline() your going to wait until a complete line is read. You need to instead use read(1).

while time.time() < end:
    if poller.poll(timeout):
        # works as expected as long as output.sh produces lines
        # read() also blocks
        print "%s" % proc.stdout.read(1),