Is is possible to somehow use Send/Recv in ZMQ simultaniousely(by multithreading)?

Question

I am trying to come up how to effectively use ZMQ to multithread (so send doesn't block receive and receive doesn't block send).

I wanted to use ZMQ_DONTWAIT flag but when sending the data, it will sometimes not be send ( EAGAIN error, so I would have to re-queue the message which is a waste of resources when dealing with megabytes of data).

I did come up with the following code:

Concurrency::concurrent_queue<zmq::message_t> QUEUE_IN;
Concurrency::concurrent_queue<zmq::message_t> QUEUE_OUT;

void SendThread(zmq::context_t &context) {
    zmq::socket_t zmq_socket(context, ZMQ_DEALER);
    zmq_socket.connect(string_format("tcp://%s:%s", address, port).c_str());
    zmq::message_t reply;
    while (true) {
        while (QUEUE_OUT.try_pop(reply))
            zmq_socket.send(reply);
        Sleep(1);
    }
}

void RecvThread(zmq::context_t &context) {
    zmq::socket_t zmq_socket(context, ZMQ_DEALER);
    zmq_socket.connect(string_format("tcp://%s:%s", address, port).c_str());
    zmq::message_t reply;
    while (true) {
        while (zmq_socket.recv(&reply))
            QUEUE_IN.push(reply);
    }
}

void ConnectionThread()
{
    zmq::context_t context(1);
    std::thread* threads[2] = { 
        new std::thread(SendThread, context), 
        new std::thread(RecvThread, context)
    };
    threads[0]->join();
}

However that would require two sockets on the server end, and I would need to identify to which I need to send data and to which I need to listen on the server end, right? Is there no way to use one socket yet use send and receive in a multithreaded environment?

I would maybe like to do it asychroniously on one socket, but after studying the async sample I still don't grasp the idea as there aren't much comments around it.

Answer 1

1. When sending large data (you say you're sending MB of data in a single message), it's going to take some time, ZMQ doesn't "duplex" sending and receiving so that they can both actually happen. The DONTWAIT flag isn't going to help you so much there, its purpose is to ensure that you're not waiting on ZMQ when you could be performing non-ZMQ actions. All messages should still be queued up in any event (barring interference from the High Water Mark)
2. The only way to safely use multiple threads to parallelize sending and receiving is to use multiple sockets.

But, it's not all bad. If you use one designated send socket and one designated receive socket, then you can use pub/sub, which opens up some interesting options.

Answer 2

Avoiding the Sleep

To avoid the sleep, you can use zmq_poll() using a ZMQ_POLLOUT event to protect the send(). You don't need to use ZMQ_DONTWAIT. [I used the C function there, your binding will have the equivalent.]

Routing to RecvThread

One cannot share sockets between threads, so 2 sockets are needed for this to work. The server would only need one socket (presumably ROUTER) that bound to 2 ports. When it receives a message, it will then need to know where to send the reply...

When a ROUTER socket receives a message, the zmq internals adds a frame to the message with the identity of the sender. This frame will be seen by the server code, which would normally use that same identity frame when constructing a message to reply to the sender. In your case, that's the client's SendThread. OTOH, you want to reply to the client's receive socket, so the identity frame must be for that.

The only thing left is how the server obtains the identity frame of the client's receive socket. For that, you'll need to invent a small protocol. Arranging for the client's RecvThread to send one message to the server would almost be enough. The server should understand that message and simply retain the identity frame of the client's receive socket, and use a copy of it when constructing reply messages.

All of this is explained in the guide under "Exploring ROUTER Sockets".