What am I doing wrong with sendline() in my twisted server?

Question

I'm trying to send the contents of a file over to the client 1 line at a time so that the client (written in Objective-C) can process each line individually. However the log of the client shows that the data being send over from the server is all coming through as 1 line and is apparently too large so the it cuts off mid way through one line causing the client to crash because of the unexpected syntax.

Is there something i'm doing on the server(written in python with twisted) that is causing the lines to not be sent separately?

Here is the particular code in the server that is holding me up at the moment.

def sendLine(self, line): 
    self.transport.write(line + '\r\n') 

def updateShiftList(self):

    #open the datesRequested file for the appropriate store and load the dates into a list
    fob = open('stores/'+self.storeName+'/requests/datesRequested','r')
    DATES_REQUESTED = fob.read()
    datesRequested = DATES_REQUESTED.split('\n')
    #open each date file that is listed in datesRequested
    for date in datesRequested:
        if os.path.isfile('stores/'+self.storeName+'/requests/' + date):
            fob2 = open('stores/'+self.storeName+'/requests/' + date,'r')
            #load the file into memory and split the individual requests up
            THE_REQUESTS = fob2.read()
            thedaysRequests = THE_REQUESTS.split('\n')
            for oneRequest in thedaysRequests:
                if len(oneRequest) > 4:
                    print "*)[*_-b4.New_REQUEST:"+oneRequest
                    self.sendLine('*)[*_-b4.New_REQUEST:'+oneRequest)
            fob2.close()
    fob.close()

So frustrating and i'm sure it's something easy. Thanks.

Answer 1

This question concerns a topic that is frequently raised. There are a number of questions on stackoverflow about the same issue:

• TCP messages getting coalesced
• gen_tcp smushed messages
• Broken TCP messages
• How to separate TCP socket messages in node.js

etc.

TCP sends an ordered, reliable stream of data.

• Ordering means that bytes sent first arrive first.
• Reliability means bytes sent will be delivered, or the connection will break. Data is not silently dropped.

Streaming is what mostly concerns your question. A stream is not divided up into separate messages. It consists of bytes, and the "boundary" between those bytes can move arbitrarily.

If you send "hello, " and then you send "world", the boundary between those two strings in the stream may disappear. The peer may receive "hello, world", or "h", "ello, world", or "he", "ll", "o,", " w", "or", "ld".

This is the very reason people use line-oriented protocols. The "\\r\\n" at the end of each logical message lets the receiver buffer and split the stream up into those original logical messages.

For a deeper dive, I recommend this video of a recent PyCon presentation: http://pyvideo.org/speaker/417/glyph

This all points towards the other end of your connection, the ObjC client, as the source of your misbehavior.