Combining tornado and zmq ioloops: Connection reset by peer exception

Question

I faced with one annoying (while not critical) problem when I tried to combine Tornado and pyzmq ioloops as described in the pyzmq official documentation .

I have a process running tornado (T) server which accepts REST API requests from clients (C) and proxies them though ZMQ transport to another process (Z) that does real work.

C <-> T <-> Z

If C closes the connection before Z replies to T, Z (tornado) outputs long bunch of exception traces (see at the bottom). Imagine the following example:

import tornado.ioloop
from tornado.web import Application, RequestHandler, asynchronous
from zmq.eventloop import ioloop
import time

def time_consuming_task():
    time.sleep(5)

class TestHandler(RequestHandler):
    def get(self, arg):
        print "Test arg", arg
        time_consuming_task()
        print "Ok, time to reply"
        self.write("Reply")

if __name__ == "__main__":
    app = tornado.web.Application(
        [
            (r"/test/([0-9]+)", TestHandler) 
        ])

    ioloop.install()
    app.listen(8080)
    tornado.ioloop.IOLoop.instance().start()

This example doesn't actually talk to any ZMQ peer, it just attaches pyzmq ioloop to tornado's ioloop. Though, it's enough to illustrate the problem.

From console one run server:

% python example.py

From console two run client and interrupt it before server replies (ie during 5 secs):

% curl -is http://localhost:8080/test/1
^C 

The output of server is:

Test arg 1
Ok, time to reply
WARNING:root:Read error on 24: [Errno 54] Connection reset by peer
ERROR:root:Uncaught exception GET /test/1 (::1)
HTTPRequest(protocol='http', host='localhost:8080', method='GET', uri='/test/1', version='HTTP/1.1', remote_ip='::1', body='', headers={'Host': 'localhost:8080', 'Accept': '*/*', 'User-Agent': 'curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5'})
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1023, in _execute
    self.finish()
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 701, in finish
    self.request.finish()
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 433, in finish
    self.connection.finish()
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 187, in finish
    self._finish_request()
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 223, in _finish_request
    self.stream.read_until(b("\r\n\r\n"), self._header_callback)
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 153, in read_until
    self._try_inline_read()
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 386, in _try_inline_read
    if self._read_to_buffer() == 0:
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 421, in _read_to_buffer
    chunk = self._read_from_socket()
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 402, in _read_from_socket
    chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 54] Connection reset by peer
ERROR:root:Cannot send error response after headers written
ERROR:root:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
    callback(*args)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
    self.request_callback(self._request)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
    handler._execute(transforms, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
    self._handle_request_exception(e)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
    self.send_error(500, exc_info=sys.exc_info())
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
    self.finish()
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
    self.flush(include_footers=True)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
    self.request.write(headers + chunk, callback=callback)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
    self.connection.write(chunk, callback=callback)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
    assert self._request, "Request closed"
AssertionError: Request closed
ERROR:root:Exception in callback 
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pyzmq-2.2.0-py2.7-macosx-10.7-intel.egg/zmq/eventloop/ioloop.py", line 434, in _run_callback
    callback()
  File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
    callback(*args)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
    self.request_callback(self._request)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
    handler._execute(transforms, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
    self._handle_request_exception(e)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
    self.send_error(500, exc_info=sys.exc_info())
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
    self.finish()
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
    self.flush(include_footers=True)
  File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
    self.request.write(headers + chunk, callback=callback)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
    self.connection.write(chunk, callback=callback)
  File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
    assert self._request, "Request closed"
AssertionError: Request closed

NOTE: It seems it's pyzmq related problem because disappears after excluding pyzmq ioloop.

Server doesn't die, it can be used by other clients, so the problem is not critical. Though, it's very annoying to find these huge confusing traces in logfiles.

So, are there any well-known methods to solve this problem? Thanks.

Answer 1

It's not ZMQ problem. Request can be closed by other than timeout reasons. ZMQ only problem here is that they are raising AssertionError , which is common, instead of more specific exception.

If you are sure, that you don't want to have these exceptions in log files - do something like this:

try: 
    time_consuming_task()
except AssertionError as e:
    if e.message == 'Request closed': 
        logging.info('Bad, annoying client, came to us again!') 
    else:
        raise e