Message queue proxy in Python + Twisted

Question

I want to implement a lightweight Message Queue proxy. It's job is to receive messages from a web application (PHP) and send them to the Message Queue server asynchronously. The reason for this proxy is that the MQ isn't always avaliable and is sometimes lagging, or even down, but I want to make sure the messages are delivered, and the web application returns immediately.

So, PHP would send the message to the MQ proxy running on the same host. That proxy would save the messages to SQLite for persistence, in case of crashes. At the same time it would send the messages from SQLite to the MQ in batches when the connection is available, and delete them from SQLite.

Now, the way I understand, there are these components in this service:

1. message listener (listens to the messages from PHP and writes them to a Incoming Queue)
2. DB flusher (reads messages from the Incoming Queue and saves them to a database; due to SQLite single-threadedness)
3. MQ connection handler (keeps the connection to the MQ server online by reconnecting)
4. message sender (collects messages from SQlite db and sends them to the MQ server, then removes them from db)

I was thinking of using Twisted for #1 (TCPServer), but I'm having problem with integrating it with other points, which aren't event-driven. Intuition tells me that each of these points should be running in a separate thread, because all are IO-bound and independent of each other, but I could easily put them in a single thread. Even though, I couldn't find any good and clear (to me) examples on how to implement this worker thread aside of Twisted's main loop.

The example I've started with is the chatserver.py , which uses service.Application and internet.TCPServer objects. If I start my own thread prior to creating TCPServer service, it runs a few times, but the it stops and never runs again. I'm not sure, why this is happening, but it's probably because I don't use threads with Twisted correctly.

Any suggestions on how to implement a separate worker thread and keep Twisted? Do you have any alternative architectures in mind?

Answer 1

You're basically considering writing an ad-hoc extension to your messaging server, the job of which it is to provide whatever reliability guarantees you've asked of it.

Instead, perhaps you should take the hardware where you were planning to run this new proxy and run another MQ node on it. The new node should take care of persisting and relaying messages that you deliver to it while the other nodes are overloaded or offline.

Answer 2

Maybe it's not the best bang for your buck to use a separate thread in Twisted to get around a blocking call, but sometimes the least evil solution is the best. Here's a link that shows you how to integrate threading into Twisted:

http://twistedmatrix.com/documents/10.1.0/core/howto/threading.html

Sometimes in a pinch easy-to-implement is faster than hours/days of research which may all turn out to be for nought.

Answer 3

A neat solution to this problem would be to use the Key Value store Redis . Its a high speed persistent data store, with plenty of clients - it has a php and a python client (if you want to use a timed/batch process to process messages - it saves you creating a database, and also deals with your persistence stories. It runs fine on Cywin/Windows + posix environments.

PHP Redis client is here .

Python client is here .

Both have a very clean and simple API. Redis also offers a publish/subscribe mechanism, should you need it, although it sounds like it would be of limited value if you're publishing to an inconsistent queue.