How to resolve Tornado + ZMQ error

Question

Like everyone else, I'm working through a chat PoC using Tornado (SockJS-Tornado) + pyzmq.

The overall idea is to have N tornado instances, each with connected websocket clients, exchanging messages through a ZMQ Device. In other words a websocket client on instance A publishes to a ZMQ Forwarder. Instances B & C have Handlers that are subscribed to the Forwarder so that matching topics are relayed to the websocket clients of B & C. And yes, the clients on B & C also have publisher sockets that publish to the Forwarder.

This all works fine but my problem is that if one of the connected clients disconnects unexpectedly (eg closes or refreshes the browser tab), then I get a deluge of errors like this:

ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File ".../tornado/ioloop.py", line 836, in start
  fd_obj, handler_func = self._handlers[fd]
KeyError: 246850817
ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File ".../tornado/ioloop.py", line 836, in start
  fd_obj, handler_func = self._handlers[fd]
KeyError: 246850817

So, at the application level, there's a single Context, but each Handler instantiated gets its own Publisher and Subscriber sockets. The Subscriber sockets use ZMQStream, like this:

   def _set_up_sockets(self):
    self.publisher = self.context.socket(zmq.PUB)
    self.publisher.connect('tcp://127.0.0.1:5559')
    self.subscriber = self.context.socket(zmq.SUB)
    self.subscriber.connect('tcp://127.0.0.1:5560')
    self.subscriber.setsockopt(zmq.SUBSCRIBE, "_NOROOM_")
    stream = zmqstream.ZMQStream(self.subscriber,
                                 io_loop=self.session.server.io_loop)
    stream.on_recv(self.echo)

The ioloop passed to ZMQStream is the global io_loop created at application startup:

ioloop.install()
io_loop = IOLoop.instance()

It's actually an instance of the ZMQ subclass of the Tornado IOLoop. Any idea what's causing this error? I have some cleanup methods, on_connection_close and on_close but they don't seem to get called at any point.

Oh, this is Python 2.7.8, ZeroMQ 4.0.4, PyZMQ 14.4.0 (same thing happened with 14.3.1).

Thanks in advance

Answer 1

Make sure you call _set_up_sockets inside the main thread.

As you only have a single IOLoop, the zmq FAQ clearly states you need to create and use sockets in the same thread.