I am attempting to receive a series of protobufs via a socket; I won't know the amount of data in advance. I'm sending a fair amount of them, and need to buffer the messages as I receive them (to ensure that I get all messages). I would like to make use of the bytearray/memoryview available in Python to eliminate an unnecessary copy.
I'm currently using a string and appending the data as I receive it. This is easy and I can "shift" down the "buffer" by doing something like:
# Create the buffer
str_buffer = []
# Get some data and add it to our "buffer"
str_buffer += "Hello World"
# Do something with the data . . .
# "shift"/offset the message by the data we processed
str_buffer = str_buffer[6:]
Is it possible to do something similar using a bytearray/memoryview?
# Create the buffer/memoryarray
buffer = bytearray(1024)
view = memoryview(buffer)
# I can set a single byte
view[0] = 'a'
# I can "offset" the view by the data we processed, but doing this
# shrinks the view by 3 bytes. Doing this multiple times eventually shrinks
# the view to 0.
view = view[3:]
The problem arises when I try to add more data to the end. If I ever "offset" the existing view, the size of the view "shrinks*" and I can add less and less data. Is there anyway to reuse the existing memoryview and just shift the data to the left?
*Based on the documentation, I know I can't resize the array. I think the illusion of shrinking is a misunderstanding on my part.
you really, honestly don't need to know in advance how much data to expect, just keep reading until you don't get any more data:
import socket, sys
HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
recvbuff = bytearray(16)
recvview = memoryview(recvbuff)
size = 0
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
while True:
nbytes = s.recv_into(recvview)
if not nbytes:
break
size += nbytes
recvview = recvview[nbytes:]
if not len(recvview):
print "filled a chunk", recvbuff
recvview = memoryview(recvbuff)
print 'end of data', recvbuff[:len(recvview)], size
s.close()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.