从字节流读取时数据损坏

Question

For a networking project, I'm using UDP Multicast to build an overlay network with my own implementation of IP.

I use the following to parse and build my Header first, then append the payload:

def __init__(buffer_size_bytes):
    self.__buffer = bytearray(buffer_size_bytes)

def read_sock(self, listening_socket):
    n_bytes, addr = listening_socket.recvfrom_into(self.__buffer, Packet.HEADER_SIZE)
    packet = Packet.parse_header(self.__buffer)

    if packet.payload_length is not 0:
        packet.payload = parse_payload(packet.payload_length, listening_socket)

    self.__router.add_to_route_queue(packet, listening_socket.locator)

def parse_payload(to_read, socket):
    payload = bytearray(to_read)
    view = memoryview(payload)
    while to_read:
        n_bytes, addr = socket.recvfrom_into(view, to_read)
        view = view[n_bytes:]
        to_read -= n_bytes

    return payload

The header seems to be parsed correctly, but the payload gets corrupted every time. I can't figure out what I'm doing wrong when parsing the payload, and I can confirm I'm sending a bytearray from the other side.

For example, when I send a packet with the payload "Hello World" encoded in utf-8, I receive the following:

b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00'

The Packet.parse_header method:

def parse_header(cls, packet_bytes):
    values = struct.unpack(cls.ILNPv6_HEADER_FORMAT, packet_bytes[:cls.HEADER_SIZE])

    flow_label = values[0] & 1048575
    traffic_class = (values[0] >> 20 & 255)
    version = values[0] >> 28
    payload_length = values[1]
    next_header = values[2]
    hop_limit = values[3]
    src = (values[4], values[5])
    dest = (values[6], values[7])

    return Packet(src, dest, next_header, hop_limit, version, traffic_class, flow_label, payload_length)

For reference, the entire sent packet looks like this:

b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01Hello World'

On receiving the first packet, the socket.recvfrom_into blocks when reading for the payload, and doesn't return until I send another message. It then seems to discard the payload of the previous message and use the second packet received as the payload...

Answer 1

Found my explanation here .

So the key thing was that I'm using UDP. And UDP sockets discard anything that doesn't fit in the buffer you give it.

TCP sockets however behave more like the bytestream I was expecting.

Fun!