Python TCP buffer overflow

Question

I have a client server communication, I wrote the following server to handle the incoming message, but If the message is bigger than the buffer than it will be lost. How can I receive the whole packages in case if the messages are bigger than the buffer size? Is there any possibility, or I have to force the client(send a message at the begging with the maximum buffer size) to send a message within the buffer size?

msg ='' 
while( True ):                 
     msg += server.recv( 20480 )                                    
     aSplit = msg.partition( "</packet>" ) 
     #We received the full message
     while( aSplit[ 1 ] == "</packet>" ):                           
          messagehandler(  aSplit[ 0 ] + "</packet>" )                        
          msg = aSplit[ 2 ]
          aSplit = msg.partition( "</packet>" )

Answer 1

When dealing with any kind of packetised message format, you only really have two choices:

1. Make sure your buffer is big enough to deal with the whole message.
2. Write your code so that it can parse partial messages.

When I say "buffer", though, I don't mean the parameter for recv() - you can make that as small as you like, and just go around your while loop multiple times until you have a whole message.

So, to take the buffering approach you could do something like this:

msg = ''
while True:
    msg += server.recv(8192)
    while True:
        aSplit = msg.partition("</packet>")
        if not aSplit[1]:
            break
        messagehandler(aSplit[0] + "</packet>")
        msg = aSplit[2]

This works because if </packet> isn't found then partition() still returns a 3-tuple where the first item is the entire string and the other two are empty. So, all the while that partition() returns a non-empty string for the separator then a packet has been found. As soon as that's empty, there's a partial packet in msg (or it's empty), so we go back to reading from the network until we get an entire packet again.

This does involve buffering up the whole message in the msg string, but that's fine unless you expect these messages to get very large (multiple megabytes) - this could happen if the messages include large files, for example. In this case you need to be more clever and do something like swap data out to disk or process the data as you receive it.

Let me know if I wasn't clear about any of that.

EDIT: I should add, it's generally a good idea to make sure the buffer (ie msg ) doesn't get too large - if it does then you need to close the connection because something's gone wrong. This stops something feeding the application endless data until the memory runs out on the system, either accidentally or maliciously. Also, you need to be very sure that the string </packet> can't actually occur inside the message - that would split the message in half incorrectly.