Unable to Send Large File Python

Question

I am trying to implement a simple epoll web server for a class. Everything works great until I try and send a large file. I noticed it sends about 2/3 of the data then just kind of sits there and does nothing. This is how I try to ensure all my data is sent:

def sendResponse(self,fd,response):
    if response:
        totalsent = 0
        MSGLEN = len(response)
        while totalsent < MSGLEN:
            sent = self.clients[fd].send(response[totalsent:])
            if sent == 0:
                raise RuntimeError("socket connection broken")
            totalsent = totalsent + sent
    else:
        self.poller.unregister(fd)
        self.clients[fd].close()
        del self.clients[fd]

Did I mention this works on small to medium sized file. I only noticed it breaks when trying to send files 1.7 Mbs or larger.

Answer 1

If you may, skip the phase of re-inventing a wheel & use ZeroMQ messaging layer.

This way your code may forget about all low-lever qritty-nitties and your dev-time & focus is dedicated to your problem-domain issues. ZeroMQ provides both a concept and a set of ready-made tools for adding (almost)-linear-scaleability, resource-pools, failure-resilience, higher-layer abstract protocols and much more.

For an initial piece of cake, taste a view from Nicholas Piel

Spending a week with Pieters Hintjens book "Code Connected, Vol.1" will introduce you to a new world of distributed processing, that you have ready on your fingertips, just ready to re-use the knowhow of The Masters.

What you get?

A devilish fast, be it a few bytes or a few GB BLOBs, low latency and much more, all from a code simple like this:

def sendResponse( self, aZmqConnectionToCounterparty, response ):
     if response:
          try:
              aZmqConnectionToCounterparty.send( response, zmq.NOBLOCK )
          except ZMQerror:
              ''' handle ZMQerror   '''
          except ValueError:
              ''' handle ValueError '''
          except TypeError:
              ''' handle TypeError  '''
    ...

PyZMQ documentation reads:

.send( data, flags = 0, copy = True, track = False )
Send a message on this socket.

This queues the message to be sent by the IO thread at a later time.

Parameters:
-----------
data:   object, str, Frame    The content of the message.
flags:  int                   Any supported flag: NOBLOCK, SNDMORE.
copy:   bool                  Should the message be sent in a copying or non-copying manner.
track:  bool                  Should the message be tracked for notification
                              that ZMQ has finished with it? ( ignored if copy = True )

Returns:
--------
None:   if copy or not track

None if message was sent, raises an exception otherwise.

MessageTracker: if track and not copy

a MessageTracker object, whose pending property will be True until the send is completed.

Raises:
-------
TypeError                    If a unicode object is passed
ValueError                   If track=True, but an untracked Frame is passed.
ZMQError                     If the send does not succeed for any reason.