How to can I serialize data to Communication between C++ zmq client and Python zmq Server

Question

UPDATING MY QUESTION

How to can I represent the arrived message in my python zmq server to show their content ?

According to this behavior, can I assume that the btnState data is sent to python server in anyway?

Context:

I am sending some data members structures using a C++ zeromq client process: ZMQComponent.h file

#include <zmq.hpp>
#include <sofa/defaulttype/VecTypes.h>

// To Quat datatype
#include <sofa/defaulttype/Quat.h>
using sofa::defaulttype::Quat;

using std::string;

namespace sofa
{

namespace component
{

namespace controller
{

/* data structure which I want send data to python zmq server */
struct instrumentData
{
  typedef sofa::defaulttype::Vec3d Vec3d;
  Vec3d pos;
  Quat quat;
  int btnState;
  float openInst;
  bool blnDataReady;
};

class ZMQComponent : public sofa::core::behavior::BaseController
{
  public:
    SOFA_CLASS(ZMQComponent, sofa::core::behavior::BaseController);

    ZMQComponent();
    virtual ~ZMQComponent();

    /* Conect to ZMQ external python Server  */
    void setupConnection();

    /* Send some data memeber instrumentData structure to ZMQ external Server  */
    void instrumentDataSend(instrumentData a);

    /* initialize function */
    void init();

};

} // namespace sofa

} // namespace component

} // namespace controller

The ZMQComponent.cpp is:

#include <sofa/core/ObjectFactory.h>
#include <zmq.hpp>
#include <iostream>
#include <string>
#include "ZMQComponent.h"


using namespace std;

namespace sofa
{

namespace component
{

namespace controller
{

/*  ZMQ Internal Client context and socket */
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REQ);

ZMQComponent::ZMQComponent(){}

void ZMQComponent::setupConnection()
{
    cout << "Connecting to python zeroMQ server ..." << endl;
    socket.connect("tcp://localhost:5555");
}

void ZMQComponent::instrumentDataSend(instrumentData a)
{
    /*  Initialize the data members structure instrumentData */
    a.pos = sofa::defaulttype::Vec3d(1.0f, 1.0f, 1.0f);
    a.quat = defaulttype::Quat(1.0f, 1.0f, 4.0f, 1.0f);
    a.btnState = 5671;
    a.openInst = 1.0f;
    a.blnDataReady = false;

    string s, test, result, d;
    s = to_string(a.btnState);
    test = " is a number";
    result = s + test;

    /*  We send  the btnState data  */
    zmq::message_t request(30);



/*  We ask for the memory address to ge the btnState content and send it. */
    memcpy(request.data(), &result, 30);
    socket.send(request);
}


/*  In the init function we create the objects to setup connection and send data  */
void ZMQComponent::init()
{
    std::cout << "ZeroMQCommunication::init()" << std::endl;
    ZMQComponent z;
    z.setupConnection();

    instrumentData itemp;
    z.instrumentDataSend(itemp);

}

/*  Other code related ....  */
ZMQComponent::~ZMQComponent(){}

// int ZeroMqComponentClass = sofa::core::RegisterObject("This component does nothing.").add<ZeroMqComponent>();
SOFA_DECL_CLASS(ZMQServerComponent)

int ZMQServerComponentClass = sofa::core::RegisterObject("This component create a Socket.").add< ZMQServerComponent >();
} // namespace controller

} // namespace component

} // namespace sofa

Then , my python zmq server which receive the btnState int variable is:

import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
print('ZMQ Server listening ... ')

while True:
    #  Wait for next request from client
    message = socket.recv()
    print("Received message from Sofa: {}".format(message))

    #  Do some 'work'
    time.sleep(1)

The output or the message which arrive to python zmq server is the content of result variable ( btnState turn to string in s content variable + string test concatenated) and some symbols characters of the :

(cnvss_test) ➜  Python git:(ZMQCommunication) ✗ python server.py
ZMQ Server listening ...
Received message from Sofa: b'\xb0\x1d\x19\xf4\xfd\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 is a number'

The previous output on my ZMQ python server script show that the string result from sofa is arrived to server, and their content is visualized, but too, this strings or characters symbols, which are product or consequence of the size of the zmq::message_t request(30) defined in my C++ client in the beginning.

If I assign a value less than 30 in the request, by example zmq::message_t request(10) the output in my server is:

Received message from Sofa: b'\x90\x94\xa1\x00\xfc\x7f\x00\x00\x0e\x00'

If I assign a value greater than 10 in the request, by example zmq::message_t request(20) the output in my server is:

Received message from Sofa: b'\x80$(\xc7\xfc\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 i

Then, the string or object which I receive in the server side, it has as long as the length or size assigned to zmq::message_t request variable

Based in the above mentioned, is ZMQ whom add this strings in my message received?

According to the previous process, my message is arrived to my server, then is correct attempt what the serialization process with some entity like protocol buffer is necessary? I understand that use something like google protocol buffer allow have some correlation more controlled in relation to objects sent and the objects received in relation of their real content ...

In any case, how to can I remove the strings or characters symbols that are added in the message arrived to the server?

Any support or orientation will be highly appreciated

Answer 1

Your system is heterogeneous, meaning you need some sort of serialisation that is platform / language agnostic.

For your purposes the most convenient one to use is likely to be Google Protocol Buffers . This supports both C++ and Python very nicely. With this you'll be defining your messages / data structures in a schema file (file extension .proto ), and using protoc to compile that to both C++ source code and also to Python source code. These give you classes that can serialise/deserialise to/from the same wireformat. The serialisation / deserialisation can nicely integrated with ZMQ message buffers.

There are others;

• Apache Avro is a possibility.
• I would avoid XSD schemas; in principle they're fine, but finding code generators that actually do a proper and complete job is difficult / expensive. For example, xsd.exe (from Microsoft) can compile an XSD schema to C++ classes (I think), but ignores the constraint fields in the schema (ed MinInclusive ).
• ASN1 is really good, but I've yet to find a decent implementation for Python. There is a code-first implementation of ASN.1 for python (pyasn), but that rather misses the whole point of having an ASN.1 schema...

Answer 2

I'm using a similar zmq implementation in c++ with the sofa-framework (related to the autor plugin).

I don't have any issue sending data from zmq C++ to python zmq.

Here is an overview of my zmq server written in python :

import zmq

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, "")

print "Collecting data from c++ ..."
socket.connect ("tcp://127.0.0.1:6000")

while True:
    print socket.recv()

An here is an overview of the result :

douaille@douaille:~/Documents/zmq$ python zmqClient.py 
Collecting data from c++ ...
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 
/colorLight string:'1 1 1 1' 

I'm using zmq as publisher, but it does not change anything for your needs. Plus the implementation for the request client is almost the same. It looks like you are printing the request and not the message

You can find an implementation of the c++ sender part here : https://github.com/SofaDefrost/sofa/blob/sofaCommunication/applications/plugins/Communication/components/serverCommunicationZMQ.inl

EDIT :

Here is an example using zmq request :

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)

print("Collecting data from c++ using REQ ...")
socket.connect("tcp://localhost:6000")

while True:
    print("Sending request")
    socket.send(b"Hello")
    message = socket.recv()
    print("Received reply : %s" % message)

Result :

douaille@douaille:~/Documents/zmq$ python zmqClient.py 
Collecting data from c++ using REQ ...
Sending request
Received reply : /colorLight string:'1 1 1 1' 
Sending request
Received reply : /colorLight string:'1 1 1 1' 
Sending request
Received reply : /colorLight string:'1 1 1 1'