如何使用ctypes多次recv_into C缓冲区？

Question

I have a Python library that is using ctypes to register a callback with a C library. The signature of the callback is:

CFUNCTYPE(c_int_32, c_uint_32, POINTER(c_byte), POINTER(c_size_t))

In this case, the third argument is a pointer to a byte array allocated by the C library, and the fourth argument is its length.

I would like to populate the byte array with data from the network by calling socket.recv_into . More importantly, I would like to completely fill this byte array: that is, if recv_into returns fewer bytes than the fourth argument of this function then I'd like to call recv_into again.

Suppose my function looks like this:

def callback(first, second, buffer, size):
    total_read = 0
    while total_read < size[0]:
        total_read += some_socket.recv_into(buffer, size[0] - total_read)
        # XXX: What happens here???

My question is: how can I manipulate the value of buffer to ensure that each call to recv_into appends to the buffer, rather than overwriting previously-written data?

Answer 1

Building on the comments, without intermediate casting:

# assuming we know OFFSET (where to start copying in the buffer)
# and AMOUNT (how many bytes we will read)
tmp_buf = (ctypes.c_char * size).from_address(ctypes.addressof(buffer.contents))
mview = memoryview(tmp_buf)[OFFSET:AMOUNT]
_ = sock.recv_into(mview, AMOUNT)

buffer.contents returns the pointer to the buffer, so its address can be extracted using ctypes

This worked for me in a simple C code that passes a pre-allocated buffer into a callback, that is registered in the python code.

Answer 2

I believe the trick is to wrap it into a memoryview:

>>> buf = ctypes.create_string_buffer(b"fooxxx")
>>> sock.recv_into(memoryview(buf)[3:],3)  # receiving b"bar"
3
>>> buf.value
b'foobar'

Answer 3

Here's a code sample that shows how to have recv_into "append" into the buffer. Lots of boilerplate around setting up a socket but this allows you to run it immediatelly and see it in action.

import socket
import ctypes


_BUFFER_SIZE = 20
BufferType = ctypes.c_byte * _BUFFER_SIZE


if __name__ == '__main__':

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(('127.0.0.1', 10000))
    s.listen(1)
    c, _ = s.accept()

    buffer = BufferType()
    offset = 0
    available = _BUFFER_SIZE
    memview = memoryview(buffer)

    while offset < _BUFFER_SIZE:
        print('recv_into: offset=%r, available=%r' % (offset, available))
        count = c.recv_into(memview[offset:], available)
        offset += count
        available -= count

    print('done: buffer=%r' % (memview.tobytes(),))

    c.close()
    s.close()

Does this solve your problem?