ZMQ: socket_send/recv blocking

Question

So I'm trying to set up some simple communication between ZMQ in Python and ZMQ in a C/C++ extension. Python sets up the context, binds an inproc socket, and passes the context and socket name to the extension. The extension sets up its own socket, connects, and listens for messages. Python then sends a header and then the string representation of a dictionary over to the extension. Pretty simple stuff using REQ/REP sockets. However, for some reason I can't seem to find, the call to socket.send is blocking, and the extension never gets past the call to zmq_recv. I have a test environment where I have almost exactly the same scenario play out, but the sockets don't block, and I've triple-checked the code and it should be working in the same way.

PYTHON:

import zmq
import cppextension
# No lectures about using threading please. I'm restricted to, in essence
# using this function because of the code base I'm working with.
from thread import start_new_thread

socket = self.zmq_context.socket(zmq.REQ)
socket_name = "inproc://agl"
socket.bind(socket_name)
t = start_native_thread(cppextension.actor,
                        (self.zmq_context, socket_name))

test_send = {"foo": 1, "bar": 2}
# BLOCKS ON THIS LINE VVVVV
socket.send("TEST", flags=zmq.SNDMORE)
socket.send(str(test_send))
socket.recv()
socket.send("STOP")

C/C++:

// Originally these used std::basic_string<Py_UNICODE> but I reverted
// back to normal std::string so I can use a JSON parsing library.
typedef string pystring;
typedef char pystring_t;

extern "C" PyObject *
actor(PyObject *self, PyObject *args) {
    PyObject *py_context, *py_connect_to;
    PyThreadState *_save;
    void *context;
    char *connect_to;
    void *socket;
    int rc;

    if(!PyArg_ParseTuple(args, "OO", &py_context, &py_connect_to)) {
        PyErr_SetString(PyExc_TypeError, "Expected two arguments (ZMQ context, name of socket to connect to)");
        return NULL;
    }
    py_context = PyObject_GetAttrString(py_context, "_handle");
    if(py_context == NULL) {
        PyErr_SetString(PyExc_TypeError, "Could not get '_handle' from context");
        return NULL;
    }
    if(!PyInt_Check(py_context)) {
        PyErr_SetString(PyExc_TypeError, "_handle was not an integer");
        return NULL;
    }
    context = (void*)PyInt_AsLong(py_context);
    connect_to = new char[PyString_Size(py_connect_to) + 1];
    strcpy(connect_to, PyString_AsString(py_connect_to));
    _save = PyEval_SaveThread();

    //
    // GIL-less operation BEGIN
    // ** WARNING: Do NOT call any functions that begin with 'Py', or touch any
    //    data structures that begin with 'Py' while in this section. It *WILL*
    //    blow up the Python interpreter.
    //
    socket = zmq_socket(context, ZMQ_REP);
    rc = zmq_connect(socket, connect_to);

    pystring TEST("TEST");
    pystring STOP("STOP");
    pystring SUCCESS("SUCCESS");
    pystring FAILURE("FAILURE");

    if(rc == 0) {
        int going = 1;
        // Should be able to hold a full megabyte of text, which should be enough
        // for any message being passed in.
        // Is there a way to query size of the incoming message...?
        char buffer[1000000];
        while(going) {
            // BLOCKS ON THIS LINE VVVVVV
            int size = zmq_recv(socket, buffer, 1000000, 0);
            if(size == -1) {
                // ERROR
                continue;
            }
            // Assume we don't get larger than 1MB of data. Should put a
            // check around this at some point, but not right now.
            buffer[size] = 0;

            pystring fullmsg(buffer);
            cout << "ZMQ RECIEVED: " << fullmsg << endl;
            if(fullmsg == TEST) {
                size = zmq_recv(socket, &buffer, 1000000, 0);
                if(size != -1) {
                    buffer[size] = 0;
                    pystring json_fullmsg(buffer);
                    cout << "ZMQ JSON: " << json_fullmsg << endl;
                    contacts.add(json_fullmsg);
                    zmq_send(socket, SUCCESS.c_str(), SUCCESS.size() + 1, 0);
                }
                else {
                    zmq_send(socket, FAILURE.c_str(), FAILURE.size() + 1, 0);
                }
            }
            else if(fullmsg == STOP) {
                going = 0;
                zmq_send(socket, SUCCESS.c_str(), SUCCESS.size() + 1, 0);
            }
        }
    }
    else {
        // ERROR
        int err = zmq_errno();
        switch(err) {
        case EINVAL:
            cout << "ZMQ CONNECT ERR: " << "Endpoint supplied is invalid" << endl;
            break;
        default:
            cout << "ZMQ CONNECT ERR: " << err << endl;
            break;
        }
    }
    zmq_close(socket);
    //
    // GIL-less operation END
    //

    PyEval_RestoreThread(_save);
    Py_INCREF(Py_None);
    return Py_None;
}

Any help figuring out what's going on here is very appreciated.

EDIT: Also note that this code will be running in an environment where gevent has monkeypatched the standard library. This is part of the reason I'm using thread.start_new_thread, because it was saved before the monkeypatching happened, and I want a real thread not a green thread.

Answer 1

two things,

because you are using req/rep in your modified version, the "send,send,recv,send..." will not work. both the send/recv MUST work in 'lock-step' manner (send,recv,send,revc.)

the ZMQ_NOBLOCK will raise an exception of EAGAIN, which may mean "the socket connection is not complete, please come back later." try placing a timer/sleep after bind and both send/recv. this is what's causing the "Resource temporarily unavailable" msg.

hope this helps

mr. onoffon

Answer 2

Not sure this would help, but here goes:

• Are you sure that send() blocks, and it is not simply a matter of never getting the reply? You could call send() with ZMQ_NOBLOCK and see if an exception is raised. If it is, then indeed send() is failing to enqueue the message.
• Have you considered PAIR sockets instead of REQ/REP ? The guide recommends this for inproc communication between threads.