如何设计包含大量数据的JMS消息

Question

I am working on designing a system that uses an ETL tool to retrieve batches of data, ie, insert/update/deletes for one or more tables, and puts them on a JMS topic to be processed later by multiple clients. Right now, each message on the topic represents a single record I/U/D and we have a special message to delimit the end of the batch. It's important to process the batches in a single transaction, so having a bunch of messages delimited by a special one is not ideal: both sessions publishing and receiving messages must be designed for multiple messages; the batch delimiter message is a messy solution (each time we receive a message we need to check if it's the last) and very error prone; the system is difficult to debug and maintain; the number of messages on the topic becomes quickly huge (up to millions).

Now, I think that the next natural step to improve the architecture is to pack all the records in a single JMS message so that when a message is received, it encompasses a single transaction, it's easy to detect failures, there are no "orphan" records on the topic, etc. I only see advantages in doing so! Now here are my questions:

• What's the best way to create such a packed message? I think my choices are StreamMessage , ByteMessage or ObjectMessage . I excluded text and map messages because the first will require text parsing, which will kill performance, and I assume the second one doesn't really seem to fit the scenario. I'm kinda leaning towards StreamMessage because it seems quite compact although it will require a lot of work writing custom serialization code (even worse for ByteMessage). Not sure about ObjectMessage, how does it perform? Is there an out of the box solution for this?
• What's the maximum size allowed per message? Could it be in the order of hundreds of KB or even few MB?

Thanks for the thoughts!

Giovanni

Answer 1

Instead of using one large message, you could use two (or more) queues, correlation ids and a message selector.

Queueing:

1. Post a notification message to "notification queue" to indicate that processing should start
2. Post command messages to "command queue" with correlation id set to notification messages message id (you can use multiple command queues, if queue depth gets too high)
3. Commit the transaction

Processing:

1. Receive the notification message from "notification queue" (eg with message driven bean)
2. Receive and process all the related messages from "command queue" using a message selector
3. Commit the transaction

Answer 2

Using bytes (eg a ByteMessage) is likely the less memory intensive.

If you manipulate Java Objects, you can use a fast and byte effective serialization/deserialization library like Kryo

We happily use Kryo in production on a messaging system, but you have plenty of alternatives such as the popular Google Protocol Buffers