[英]Celery, periodic task execution, with concurrency
I would like to launch a periodic task every second but only if the previous task ended (db polling to send task to celery). 我想每秒钟启动一次定期任务,但前提是前一个任务结束(db轮询将任务发送到celery)。 In the Celery documentation they are using the Django cache to make a lock.
在Celery文档中,他们使用Django缓存进行锁定。
I tried to use the example: 我尝试使用示例:
from __future__ import absolute_import
import datetime
import time
from celery import shared_task
from django.core.cache import cache
LOCK_EXPIRE = 60 * 5
@shared_task
def periodic():
acquire_lock = lambda: cache.add('lock_id', 'true', LOCK_EXPIRE)
release_lock = lambda: cache.delete('lock_id')
a = acquire_lock()
if a:
try:
time.sleep(10)
print a, 'Hello ', datetime.datetime.now()
finally:
release_lock()
else:
print 'Ignore'
with the following configuration: 具有以下配置:
app.conf.update(
CELERY_IGNORE_RESULT=True,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
CELERYBEAT_SCHEDULE={
'periodic_task': {
'task': 'app_task_management.tasks.periodic',
'schedule': timedelta(seconds=1),
},
},
)
But in the console, I never see the Ignore
message and I have Hello
every second. 但是在控制台中,我从未看到“
Ignore
消息,并且每秒都有“ Hello
。 It seems that the lock is not working fine. 看来锁不能正常工作。
I launch the periodic task with: 我使用以下命令启动定期任务:
celeryd -B -A my_app
and the worker with: 和工人:
celery worker -A my_app -l info
Could you please correct my misunderstanding? 你能纠正我的误会吗?
From the Django Cache Framework documentation about local-memory cache : 从有关本地内存缓存的Django Cache Framework文档中:
Note that each process will have its own private cache instance, which means no cross-process caching is possible.
请注意,每个进程都有其自己的专用缓存实例,这意味着不可能进行跨进程缓存。
So basically your workers are each dealing with their own cache. 因此,基本上每个工作人员都在处理自己的缓存。 If you need a low resource cost cache backend I would recommend File Based Cache or Database Cache, both allow cross-process.
如果您需要低资源成本的缓存后端,我建议您使用基于文件的缓存或数据库缓存,两者都允许跨进程。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.