How to setup a clustering of a queue servers for beanstalkd?

Question

We've two Queue Servers, both attached to the application. so far Server 1 receives all the queued jobs and processed it. I would like to set up a Cluster so that the load is spread across 2 servers. anyone can suggest how to setup a cluster.

Thanks.

Answer 1

Beanstalkd doesn't offer this feature.

Alternatives are:

• you setup a soft sharding to route requests to queue A or B
• you can use alternatives like Redis Queue, Cloud Pub/Sub from Google Cloud Platform

Answer 2

Beanstalkd in a single instance setup can support multiple TCP connections, and generally outperforms Redis. Below are a few benchmarks, however benchmarks are subjective.

Benchmarks References

• https://ph4r05.deadcode.me/blog/2017/12/16/laravel-queueing-benchmark.html
• https://adam.herokuapp.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/

So a vertical scaling is usually sufficient.

The problem however, is availability, when the single beanstalkd instance goes away.

You can checkout coolbeans, this project is in alpha. It provides a replicated beanstalkd https://github.com/1xyz/coolbeans