Start celery workers with multiple brokers

Question

Currently, i have a celery.py with a single redis broker

proj/celery.py

from __future__ import absolute_import

from kombu import Exchange, Queue
from celery import Celery

app = Celery('proj',
             broker='redis://myredis.com'
             backend='redis://myredis.com'
             include=['proj.tasks])

if __name__ == '__main__':
    app.start()

i would start a worker with:

celery multi start somename -A proj -Q work -c20 --pidfile='somepidfile' --logfile='somelogfile'

over multiple machines (let's say 20..)

so these workers across 20 machines use a single broker,

'redis://myredis.com'

I would like to split that so that 10 machines use 'redis://myredis.com' and other 10 machines use 'redis://myredis2.com'

What changes do I need to make that happen?

Thank you

Answer 1

A simple way to do that will be to have a DNS that resolves to both redis://myredis.com and redis://myredis2.com in a round-robin fashion. Provided that you have enough workers, this should roughly split the 2 brokers evenly.