[英]Celery: how to limit number of tasks in queue and stop feeding when full?
I am very new to Celery and here is the question I have: 我对Celery很新,这是我的问题:
Suppose I have a script that is constantly supposed to fetch new data from DB and send it to workers using Celery. 假设我有一个脚本经常被认为是从DB获取新数据并使用Celery将其发送给工作人员。
tasks.py tasks.py
# Celery Task
from celery import Celery
app = Celery('tasks', broker='amqp://guest@localhost//')
@app.task
def process_data(x):
# Do something with x
pass
fetch_db.py fetch_db.py
# Fetch new data from DB and dispatch to workers.
from tasks import process_data
while True:
# Run DB query here to fetch new data from DB fetched_data
process_data.delay(fetched_data)
sleep(30);
Here is my concern: the data is being fetched every 30 seconds. 这是我的担忧:数据每30秒获取一次。 process_data() function could take much longer and depending on the amount of workers (especially if too few) the queue might get throttled as I understand.
process_data()函数可能需要更长的时间,并且取决于工作者的数量(特别是如果太少),队列可能会受到我所理解的限制。
The question is how do I set queue size and how do I know it is full? 问题是如何设置队列大小以及如何知道它已满? In general, how to deal with this situation?
一般来说,如何应对这种情况?
You can set rabbitmq x-max-length
in queue predeclare using kombu 您可以设置RabbitMQ的
x-max-length
使用队列预先声明海带
example : 例如:
import time
from celery import Celery
from kombu import Queue, Exchange
class Config(object):
BROKER_URL = "amqp://guest@localhost//"
CELERY_QUEUES = (
Queue(
'important',
exchange=Exchange('important'),
routing_key="important",
queue_arguments={'x-max-length': 10}
),
)
app = Celery('tasks')
app.config_from_object(Config)
@app.task(queue='important')
def process_data(x):
pass
rabbitmqctl set_policy Ten "^one-meg$" '{"max-length-bytes":1000000}' --apply-to queues
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.