基于队列文件的持久性-WebSphere MQ与SQLite队列（WAL）

Question

I use Websphere MQ and it comes by default with a "file based queue". So each queue typically has ONE physical file on disk.

So lets say for this Websphere MQ queue we have multiple readers and writers....I assume there will be some "millisecond locking" overhead between reads and writes, yet I guarantee it will still be quicker than using a full database like DB2 etc.

Now I have implemented a queue using SQLite and it works in Production happily in WAL mode.

My question is do you really think there is a huge difference in SPEED between the file based queue used by products like WebSphere MQ or the SQLite based queue? Both are file based after all and the fact that SQLIte is pure ANSI C does it well for speed.

SQLIte does offer some extra functionality like "update hooks" etc....and numerous others.

I guess what I would like to know is if you were going to implement a high speed persistent queue in C would you do it in a similar way to WebSphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode?

Thanks for the help ;-)

Answer 1

With one message producer, WMQ 7 with default logging parameters (double or triple circular logging) will store about 300-400 persistent message per second on my laptop 7200 hd drive using mqjms api while using as fast jms code as one can realistically expect (one session, one message producer, buffering on channels) and small message size (bytes message with payload size < 1k). In this scenario HDD is bottleneck.

You will get get faster speed by using lesser loging on QM side. This scales pretty linearly. Using unpersistent messages also helps.

Other common problem is network layer. Messaging is usually about sending small messages no more than few kb in size. With common network code (open connection/open session/open producer/and back) from one user you will hit TCP limitations before HDD kicks in. To avoid this problems WMQ allows message batching on channels.

I doubt that storing persistent messages in WMQ with default loging is faster than inserting blob in a DB2. Underlying mechanism is mostly the same (transactional logs, rec/ack, ...).

I never tried to compare it to SQLLite.

WMQ is not faster-than-bullet messaging solution. It is also not the easiest solution to manage. Thus said, it has bright sides and by my opinion it is one of better products out there.

Answer 2

The more constraints you place on the example, the more you would probably be able to make the numbers look like whatever you wanted. A single queue with one app? If that's your only requirement then you have lots of choices.

But take a look at some of the restrictions on WAL mode. Checkpoints wait for readers to complete. Therefore more readers you have the harder it is to checkpoint and the more disruptive checkpointing becomes. If the WAL file gets large then read performance degrades so you cannot do lazy checkpointing on a busy system and remain performant.

So the question " if you were going to implement a high speed perstent queue in C would you do it in a similar way to Websphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode? " doesn't capture all the requirements or considerations. As the number of concurrent users scales, the process architecture begins to overshadow the differences in storage approaches you are asking about. WMQ can handle thousands of concurrent readers and writers, with very large log files with fairly flat performance curves whereas the WAL docs state that performance is proportional to the size of the WAL file, ie performance curves trend down as the WAL gets large.

If I were going to implement a high speed queue in C? If you mean which existing product do I pick, I'd go for the one that has had twenty years of tuning and optimization with millions of dollars in R&D invested each of those years and which focuses exclusively on queuing. If by "implement" you mean how would I do it if I were to write a new queuing engine from scratch, then I'd be heavily influenced by the products that have focused on queuing for a really long time and try to reuse their solutions to the non-trivial problems. Databases and queuing engines didn't arrive at their respective storage architectures by accident. One is optimized for queues and one is optimized for database semantics and if you expand your scope to include DBs and queuing engines other than WMQ and SQLite this holds generally true across the categories.

Full disclosure: I've been working with WMQ for most of the twenty years it's been around and recently joined its product management team at IBM. I might be a bit biased but I've tried to focus on the technology here and not a knee-jerk "my product is better than your product" reaction. Feel free to signify agreement with up-votes, disagreement with down-votes. I'll withdraw the answer if it gets too many down-votes.