Perl Distributed Job Queue

Question

I need to setup a job queue where long running job queues are scheduled via a web interface. And are then run on multiple boxes. I also need the ability to remove and reorder jobs in the queue.

Currently all our code is in perl. I was looking in to using something like ApacheMQ, Gearman, Resque, etc...

The question, Would one of the ones mentioned before be better suitable for the job, what about a different software solution? I was originality planning just writing a MySQL/Perl queue, but that's quite difficult. I would prefer something with a decent web interface, and have an interface into Perl. A pure MySQL/Perl would be ideal.

Thanks

Answer 1

Yes, writing a good one yourself is quite difficult. Here's what I can tell you from my experience:

TheSchwartz on CPAN is a good reliable job queue that uses mysql as a backing store.

Gearman is an unreliable but low-latency queue, with a different feature set.

Neither of those have a web interface, but they have good command-line interfaces to see what's being worked on and what the state is of the queues.

You can use an MQ message queue as a solution, but those are message-passing queues, not quite the same as job management, and you'll have to handle things like scheduling and retries and failures yourself.

I've seen ActiveMQ accessed through Perl via STOMP tried and fail in two projects now.

RabbitMQ is super-stable (go, Erlang!) and quite effective.

Both of those have nice web interfaces for the message queuing part.