Interoperability Azure Service Bus Message Queue Messages

Question

I have a Java App and a NodeJS App both using a single Azure Service Bus Message Queue.

I witness some strange effects with my clients, as follow.

JAVA MESSAGE PRODUCER (using QPID libraries per Azure JMS tutorial):

 TextMessage message = sendSession.createTextMessage();
        message.setText("Test AMQP message from JMS");
        long randomMessageID = randomGenerator.nextLong() >>>1;
        message.setJMSMessageID("ID:" + randomMessageID);
        sender.send(message);
        System.out.println("Sent message with JMSMessageID = " + message.getJMSMessageID());

OUTPUT: Sent message with JMSMessageID = ID:2414932965987073843

NODEJS MESSAGE CONSUMER:

serviceBus.receiveQueueMessage(queue, {timeoutIntervalInS: timeOut, isReceiveAndDelete: true}, function(err, message) {
if(message !==null)console.log(util.inspect(message, {showHidden: false, depth: null}));
});

OUTPUT:

{ body: '@\u0006string\b3http://schemas.microsoft.com/2003/10/Serialization/�\u001aTest AMQP message from JMS',
brokerProperties:
{ DeliveryCount: 1,
EnqueuedSequenceNumber: 5000004,
EnqueuedTimeUtc: 'Wed, 04 Nov 2015 21:28:21 GMT',
MessageId: '2414932965987073843',
PartitionKey: '89',
SequenceNumber: 59672695067659070,
State: 'Active',
TimeToLive: 1209600,
To: 'moequeue' },
contentType: 'application/xml; charset=utf-8' }

If I compare that to a message inserted into the queue via serviceBus.sendQueueMessage(), then the properties look like this:

{ body: 'test message',
brokerProperties:
{ DeliveryCount: 1,
EnqueuedSequenceNumber: 0,
EnqueuedTimeUtc: 'Wed, 04 Nov 2015 21:44:03 GMT',
MessageId: 'bc0a3d4f-15ba-434f-9fb0-1a3789885f8c',
PartitionKey: '734',
SequenceNumber: 37436171906517256,
State: 'Active',
TimeToLive: 1209600 },
contentType: 'text/plain',
customProperties:
{ message_number: 0,
sent_date: Wed Nov 04 2015 21:44:03 GMT+0000 (UTC) } }

So content type is different to start with - why? - and then where does the strange garbage in the body of the first message payload come from: @ string\b3http://schemas.microsoft.com/2003/10/Serialization/� Is that the result of serialization? How can this be mitigated?

Find the code as well here: http://pastebin.com/T9RTFRBk

Answer 1

The Azure Service Bus supports two different protocols: AMQP and HTTP. The Java/JMS using qpid libs is using AMQP protocal for ServiceBus. However, the ServiceBus REST APIs wrapped in NodeJS thur HTTP protocol.

Details for AMQP support in Service Bus, please refer to https://azure.microsoft.com/en-us/documentation/articles/service-bus-amqp-overview/ .

And for REST APIs of ServiceBus, please refer to https://msdn.microsoft.com/en-us/library/azure/hh780717.aspx .

AMQP is a binary, application layer protocol, designed to efficiently support a wide variety of messaging applications and communication patterns. - from WikiPedia

But the HTTP is a text protocol.

The message format is as below, please refer to the section Message Format of the aritifact http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#section-message-format . And the AMQP specification can be refered to http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-overview-v1.0-os.html .

                                                     Bare Message
                                                            |
                                      .---------------------+--------------------.
                                      |                                          |
 +--------+-------------+-------------+------------+--------------+--------------+--------+
 | header | delivery-   | message-    | properties | application- | application- | footer |
 |        | annotations | annotations |            | properties   | data         |        |
 +--------+-------------+-------------+------------+--------------+--------------+--------+
 |                                                                                        |
 '-------------------------------------------+--------------------------------------------'
                                             |
                                      Annotated Message

So the messages sent in Java or sent in NodeJS were serialized to different results.

The content \\uXXXX formated in the content of body from AMQP is Unicode Charater.

The Unicode Charater \ is Acknowledge controll charater, please refer to https://en.wikipedia.org/wiki/Acknowledge_character to know it.

And the Unicode Charater \ is Substitute controll charater, please refer to https://en.wikipedia.org/wiki/Substitute_character .

They are limit the start and end of the metadata in the message header.

Answer 2

We have encountered the exact same issue, though in a somewhat more involved example using a Camel based producer. Due to changes in our environment we started to encounter these issues.

The issue here is how the REST service is interpreting the JMS message when it is encoding the HTTP response to the node client.

We've found that JmsTextmessage for some reason, not fully clear, are assumed to be of type "application/xml" and the content will be forwarded as such. Hence the OUTPUT you get in your example.

If instead using a JmsByteMessage, the content is interpreted as "application/octet-stream" and not mangled in transfer.

So try something along the lines of:

BytesMessage message = sendSession.createBytesMessage();
String body = "Test AMQP message from JMS";
message.writeBytes(body.getBytes(StandardCharsets.UTF_8));
sender.send(message);

We use this in order to transfer JSON encoded data to be interpreted by a Node.js client.

Answer 3

I had the same issue with the message body being prefixed by this "@\string\\b3http://schemas.microsoft.com/2003/10/Serialization/\
".

I was using Nodejs, with the azure-iot-device-mqtt and azure-iot-device packages, to send messages to IoT hub. I was using a Stream Analytics Job to receive the messages from IoT hub and publish them to a queue. I was using Nodejs, with the amqp10 package, to receive the events from the queue.

The problem was not caused by the way I was sending or receiving the messages. Rather, the problem was with the Stream Analytics compatibility level! Compatibility level 1.0 (the default at least when I deployed) uses the DataContractSerializer which serializes the message into an XML stream! Microsoft changed (fixed) this with compatibility level 1.1. So you may just need to change the compatibility level (CONFIGURE->Compatibility level) for your Stream Analytics job to 1.1.

See: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-compatibility-level#major-changes-in-the-latest-compatibility-level-11 :

Answer 4

The message sent by JMS is actually in .NET Binary XML Format (MC-NBFX), as specified here . I think this is what you get when you try to serialise an object using a DataContractSerializer with XmlDictionaryWrite.CreateBinaryWriter (although I know nothing about .NET).

Quoting the message body from above for quick reference, and replacing Unicode characters with their hex equivalent (eg to \x06 ) and \b (backspace escape) with its ASCII code 0x08 (BS):

@\x06string\x083http://schemas.microsoft.com/2003/10/Serialization/�\x1aTest AMQP message from JMS

• Key to record types is here .
• The initial @ character (ASCII code 0x40) is the type byte for the start of a ShortElement record . This is followed by the element name and its attributes.
• The ASCII code 0x06 is the start of the name, encoded as a variable-length String . This is the length byte; the string has length 6 and is the word string .
• It is followed (in the ShortElement) by zero or more Attribute Records , which start with the type byte 0x05. There are none, so the element has no attributes.
• The next byte is 0x08, which is the start of a ShortXmlnsAttribute record. The value should be the XML namespace of the <string> element, and indeed it is, again encoded as a variable-length string, with the length byte 3 (0x33 or decimal 51) and 51 characters of URL.
• The next byte is an unprintable Unicode character in the question text, and I'm not sure it's been copied correctly. In my application (where I receive similar messages from an outside system) I get the byte 0x98 instead, which is invalid in Unicode, and is the type byte of a Chars8Text record, so I'm going to assume it was one of these. In this record type, the type byte is followed by a length byte, in this case 0x1a (26 decimal), and then that number of bytes (ie 26 bytes) of text encoded as UTF-8: Test AMQP message from JMS .
• This is followed by type byte 0x01, which is an EndElement record, representing the end of the string element, ie </string> in XML text.

So the equivalent XML text would be this:

<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">Test AMQP message from JMS</string>

There are libraries that can decode this for you, for example libnbfx (C++) or msbin (Python).