How to wake up threads with ZeroMQ

Question

My current program is receiving messages via zmq. A thread is waking up from time to time and checks what zmq received last.

Now I want to improve this setup and actually process every message that comes in, instead of just the last one when waking up. Also I want my thread just to wake up, when there is something new. Is there a way how zmq can force my thread to wake up, as soon as he has a new message for my thread.

Answer 1

ZMQ is all about Actor Model programming. It is a Reactor. You use zmq_poll() to wait for one of a set of sockets to become ready for reading. When one does, you read from it.

So your program becomes a loop, at the top you call zmq_poll() which blocks (or times out), and after that you deal with whichever socket has become ready to read, and then loop.

In contrast, call backs belong in Proactor Model programming. You say what should happen up front (by setting up the call back), and then that will happen no matter what.

The important thing to remember is, Reactor and Proactor really, really don't mix. Never try to blend the two together if you can help it, it will be the source of endless confusion. Either go all in with one or the other. This is why ZeroMQ doesn't have a call back mechanism.

The poor mixing of the two styles is the reason why the IPC transport for ZeroMQ doesn't work on Windows. ZeroMQ's reactor is zmq_poll() . Fundamentally this relies on the OS providing a reactor. On Linux this is select() or epoll() which as is standard for a *nix can work with any file descriptor (network sockets, IPC pipes, serial ports, etc).

In contrast Windows does not provide a universal reactor. The closest it gets is select() , which works only on network sockets. To block on a pipe in Windows your using calls that take call backs. Windows is, fundamentally, a proactor system and there's not a lot to be done about it. This is why things like Boost.Asio is also proactor - it'll work on Windows.

It is relatively straight forward to synthesise a proactor on top of a reactor system. Your reactor loop simply becomes an event dispatcher.

In contrast attempts at doing efficient reactors on Windows have to resort to polling threads. This is what cygwin does. Their implementation of POSIX select() runs a thread per file descriptor (which are just abstractions for Windows HANDLEs), each one busy polling their handle waiting to see if data can be read. Not efficient.

Now the really, really interesting thing is that Microsoft have now gone and done a Linux runtime for Windows 10. This is a system call level implementation (so radically different to cygwin). That does support the system calls required for epoll() , select() , and it does work for pipes, sockets, etc. What I really, really want to know is how have they achieved this? Answers on a postcard please...

Answer 2

As you have mentioned " Also I want my thread just to wake up, when there is something new. ", this can be achieved if a callback is assigned to data receive , so whenever a new data arrives the callback would be executed.

But after reading about the zeromq and referring the following links:

Is it possible to add an event handling to ZeroMQ to act when data is received/sent?

Does ZeroMQ have a notification/callback event/message for when data arrives?

It looks like as of now, ZeroMQ does not implement any callbacks that would notify us when a new data is received . You can definitely check for the new data by using zmq_recv in blocking mode (but this would not mean waking up when a new data comes, it would just keep blocking until the data comes).

One possible solution as stated in the above links:

You can use SIGNALS along with the zmq which will notify whenever a new data is arrived. The advantage of using signal handler/interrupt handler is that they would not block your code until a new data arrives ( like zmq_recv() in a blocking mode ), instead signal handlers would wake up and only be executed whenever a particular signal is received.

Signals like SIGUSR1 / SIGUSR2 can be used as an indication of new data arrival. Assign the signal handler to SIGUSR1 to execute the code when a new data is arrived.

To wake up a process whenever you have a new data, you would then have to send the data as well as send a signal like:

zmq_send(...);

kill(pid, SIGUSR1);         /*  Send signal SIGUSR1 to the process
                                            whose PID is specified
                                                                   */

The receiving process receives the signal SIGUSR1 and wakes up to execute its handler. In the handler for this signal, you can read the data and perform some task. Now, this handler would only be executed again if you send some new data along with SIGUSR1 signal.