How to Log Receive Message in ZMQ Proxy?

Question

In ZMQ Proxy, we have 2 types of sockets, DEALER and ROUTER. Also, I've tried to use the capture socket, but it didn't work based on what exactly I looked for.

I'm looking for a way to log what message my proxy server receives.

Answer 1

Q : a way to log what message my proxy server receives.

The simplest way is to make use of an API v4+ directly supported logging via a ManInTheMiddle-"capture" socket:

// [ROUTER]--------------------------------------+++++++
//                                               |||||||
// [DEALER]---------------*vvvvvvvv             *vvvvvvv
int zmq_proxy (const void *frontend, const void *backend, const void *capture);
// [?]---------------------------------------------------------------*^^^^^^^

Where the capture ought be either of { ZMQ_PUB | ZMQ_DEALER | ZMQ_PUSH | ZMQ_PAIR } { ZMQ_PUB | ZMQ_DEALER | ZMQ_PUSH | ZMQ_PAIR }

If the capture socket is not NULL , the proxy shall send all messages, received on both frontend and backend , to the capture socket.

If this ZeroMQ API-granted is not meeting your expectation, feel free to express your expectations in as sufficiently detailed manner as needed ( and implement either an "external" capture -socket payload { message-content | socket_monitor() }-based filtering or one may design a brand new, user-defined logging-proxy, where your expressed features will get implemented with a use of your custom use-case specific requirements, implemented in your application-specific code, resorting to re-use but the clean and plain ZeroMQ API for all the DEALER -inbound/outbound- ROUTER message-passing and log-filtering/processing logic. )

There is no other way I can imagine to take place and solve the task.

Answer 2

It also works with a pair of PAIR sockets. As soon as one end of a pair of sockets is connected to the capture socket, messages are sent to the capture sockets AND to the other end of the proxy.

http://zguide.zeromq.org/page:all#ZeroMQ-s-Built-In-Proxy-Function and http://api.zeromq.org/3-2:zmq-proxy and http://zguide.zeromq.org/page:all#Pub-Sub-Tracing-Espresso-Pattern

helped me.

This code in python demonstrates it:

    import zmq, threading, time
    
    def peer_run(ctx):
        """ this is the run method of the PAIR thread that logs the messages
        going through the broker """
        sock = ctx.socket(zmq.PAIR)
        sock.connect("inproc://peer") # connect to the caller
        sock.send(b"") # signal the caller that we are ready
        while True:
            try:
                topic = sock.recv_string()
                obj = sock.recv_pyobj()
            except Exception:
                topic = None
                obj = sock.recv()
            print(f"\n !!! peer_run captured message with topic {topic}, obj {obj}. !!!\n")
    
    def proxyrun():
        """ zmq broker run method in separate thread because zmq.proxy blocks """ 
        xpub = ctx.socket(zmq.XPUB)
        xpub.bind(xpub_url)
        xsub = ctx.socket(zmq.XSUB)
        xsub.bind(xsub_url)
        zmq.proxy(xpub, xsub, cap)
    
    def pubrun():
        """ publisher run method in a separate thread, publishes 5 messages with topic 'Hello'"""
        socket = ctx.socket(zmq.PUB)
        socket.connect(xsub_url)
        for i in range(5):
            socket.send_string(f"Hello {i}", zmq.SNDMORE)
            socket.send_pyobj({'a' : 123})
            time.sleep(0.01)
    
    ctx = zmq.Context()
    xpub_url = "ipc://xpub"
    xsub_url = "ipc://xsub"
    #xpub_url = "tcp://127.0.0.1:5567"
    #xsub_url = "tcp://127.0.0.1:5568"
    # set up the capture socket pair
    cap = ctx.socket(zmq.PAIR)
    cap.bind("inproc://peer")
    cap_th = threading.Thread(target=peer_run, args=(ctx,), daemon=True)
    cap_th.start()
    cap.recv() # wait for signal from peer thread
    print("cap received message from peer, proceeding.")
    # start the proxy
    th_proxy=threading.Thread(target=proxyrun, daemon=True)
    th_proxy.start()
    # create req/rep socket just to prove that pub/sub can run alongside it
    zmq_rep_sock = ctx.socket(zmq.REP)
    zmq_rep_sock.bind("ipc://ghi")
    # create sub socket and connect it to proxy's pub socket
    zmq_sub_sock = ctx.socket(zmq.SUB)
    zmq_sub_sock.connect(xpub_url)
    zmq_sub_sock.setsockopt(zmq.SUBSCRIBE, b"Hello")
    # create the poller
    poller = zmq.Poller()
    poller.register(zmq_rep_sock, zmq.POLLIN)
    poller.register(zmq_sub_sock, zmq.POLLIN)         
    # create publisher thread and start it
    th_pub = threading.Thread(target=pubrun, daemon=True)
    th_pub.start()
    # receive publisher's messages ordinarily
    while True:
        events = dict(poller.poll())
        print(f"received events: {events}")
        if zmq_rep_sock in events:
            message = zmq_rep_sock.recv_pyobj()
            print(f"received zmq_rep_sock {message}")
        elif zmq_sub_sock in events:
            topic = zmq_sub_sock.recv_string()
            message = zmq_sub_sock.recv_pyobj()
            print(f"received zmq_sub_sock {topic} , {message}")

output

cap received message from peer, proceeding.

 !!! peer_run captured message with topic None, obj b'\x80\x03}q\x00X\x01\x00\x00\x00aq\x01K{s.'. !!!

received events: {<zmq.sugar.socket.Socket object at 0x76310f70>: 1}
received zmq_sub_sock Hello 1 , {'a': 123}

 !!! peer_run captured message with topic Hello 2, obj {'a': 123}. !!!

received events: {<zmq.sugar.socket.Socket object at 0x76310f70>: 1}
received zmq_sub_sock Hello 2 , {'a': 123}

 !!! peer_run captured message with topic Hello 3, obj {'a': 123}. !!!

received events: {<zmq.sugar.socket.Socket object at 0x76310f70>: 1}
received zmq_sub_sock Hello 3 , {'a': 123}

 !!! peer_run captured message with topic Hello 4, obj {'a': 123}. !!!

received events: {<zmq.sugar.socket.Socket object at 0x76310f70>: 1}
received zmq_sub_sock Hello 4 , {'a': 123}

Be aware of the slow joiner problem, hence the sleep command in the publisher/