How to go about improving this coroutine

Question

I have this co-routine, which is designed to accept data, and then send the data on to next co-routine in a chain, but in blocks of length blocksize . Due to strings being immutable I think the string appending I'm doing is pretty inefficient as a new string object will be created on every append.

As this is just 'glue' in between some proper work in these chains, it'd be nice to be able to get it as slick as possible.

def chunker(target, blocksize=DEFAULT_BLOCK_SIZE):
    buffer = ""
    target_send = target.send
    while True:
        try:
            input_data = yield
            buffer += input_data  # creates new string object every time
            buffer_len = len(buffer)
            if buffer_len >= blocksize:
                chunks, leftover = divmod(buffer_len, blocksize)
                for i in xrange(0, chunks*blocksize, blocksize):
                    target_send(buffer[i:i+blocksize])
                buffer = buffer[-leftover:] if leftover else ""
        except CleanUp:
            if buffer:
                target_send(buffer)
            target_send("")

How can I improve this? Or better, is there a simpler way of achieving this?

Answer 1

One option would be to maintain a list of each chunk and then ''.join() them when you have reached blocksize , which should be more efficient than the string concatenation. For example (untested):

def chunker(target, blocksize=DEFAULT_BLOCK_SIZE):
    data = []
    buffer = ''
    buffer_len = 0
    target_send = target.send
    while True:
        try:
            input_data = yield
            data.append(input_data)
            buffer_len += len(input_data)
            if buffer_len >= blocksize:
                buffer = ''.join(data)
                chunks, leftover = divmod(buffer_len, blocksize)
                for i in xrange(0, chunks*blocksize, blocksize):
                    target_send(buffer[i:i+blocksize])
                buffer = buffer[-leftover:] if leftover else ""
                buffer_len = len(buffer)
                data = [buffer] if buffer else []
        except CleanUp:
            buffer = ''.join(data)
            if buffer:
                target_send(buffer)
            target_send("")