[英]Topic Exchange with Celery and RabbitMQ
我對我的配置應該是什么樣的設置主題交換感到困惑。
http://www.rabbitmq.com/tutorials/tutorial-five-python.html
這就是我想要完成的事情:
Task1 -> send to QueueOne and QueueFirehose
Task2 -> sent to QueueTwo and QueueFirehose
然后:
Task1 -> consume from QueueOne
Task2 -> consume from QueueTwo
TaskFirehose -> consume from QueueFirehose
我只希望Task1從QueueOne和Task2使用來從QueueTwo中使用。
現在的問題是,當Task1和2運行時,它們也會耗盡QueueFirehose,而TaskFirehose任務永遠不會執行。
我的配置有問題,還是我誤解了什么?
CELERY_QUEUES = {
"QueueOne": {
"exchange_type": "topic",
"binding_key": "pipeline.one",
},
"QueueTwo": {
"exchange_type": "topic",
"binding_key": "pipeline.two",
},
"QueueFirehose": {
"exchange_type": "topic",
"binding_key": "pipeline.#",
},
}
CELERY_ROUTES = {
"tasks.task1": {
"queue": 'QueueOne',
"routing_key": 'pipeline.one',
},
"tasks.task2": {
"queue": 'QueueTwo',
"routing_key": 'pipeline.two',
},
"tasks.firehose": {
'queue': 'QueueFirehose',
"routing_key": 'pipeline.#',
},
}
假設你實際上意味着這樣的事情:
Task1 -> send to QueueOne
Task2 -> sent to QueueTwo
TaskFirehose -> send to QueueFirehose
然后:
Worker1 -> consume from QueueOne, QueueFirehose
Worker2 -> consume from QueueTwo, QueueFirehose
WorkerFirehose -> consume from QueueFirehose
這可能不完全是你的意思,但我認為它應該涵蓋許多場景,並希望你的。 這樣的事情應該有效:
# Advanced example starting 10 workers in the background:
# * Three of the workers processes the images and video queue
# * Two of the workers processes the data queue with loglevel DEBUG
# * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
-Q default -L:4,5 DEBUG
有關更多選項和參考: http : //celery.readthedocs.org/en/latest/reference/celery.bin.multi.html
這直接來自文檔。
我也有類似的情況,我以稍微不同的方式處理它。 我不能用芹菜多用supervisord。 因此,我在每個工人的supervisord中創建了多個程序。 無論如何,工人們將處於不同的流程中,所以只需讓主管為您處理一切。 配置文件看起來像: -
; ==================================
; celery worker supervisor example
; ==================================
[program:Worker1]
; Set full path to celery program if using virtualenv
command=celery worker -A proj --loglevel=INFO -Q QueueOne, QueueFirehose
directory=/path/to/project
user=nobody
numprocs=1
stdout_logfile=/var/log/celery/worker1.log
stderr_logfile=/var/log/celery/worker1.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
同樣,對於Worker2和WorkerFirehose,編輯相應的行以進行:
[program:Worker2]
; Set full path to celery program if using virtualenv
command=celery worker -A proj --loglevel=INFO -Q QueueTwo, QueueFirehose
和
[program:WorkerFirehose]
; Set full path to celery program if using virtualenv
command=celery worker -A proj --loglevel=INFO -Q QueueFirehose
將它們全部包含在supervisord.conf文件中,應該這樣做。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.