[英]Multiple instances of celerybeat for autoscaled django app on elasticbeanstalk
I am trying to figure out the best way to structure a Django app that uses Celery to handle async and scheduled tasks in an autoscaling AWS ElasticBeanstalk environment. 我试图找出构建Django应用程序的最佳方法,该应用程序使用Celery在自动缩放AWS ElasticBeanstalk环境中处理异步和计划任务。
So far I have used only a single instance Elastic Beanstalk environment with Celery + Celerybeat and this worked perfectly fine. 到目前为止,我只使用了Celery + Celerybeat的单个实例Elastic Beanstalk环境,这非常好用。 However, I want to have multiple instances running in my environment, because every now and then an instance crashes and it takes a lot of time until the instance is back up, but I can't scale my current architecture to more than one instance because Celerybeat is supposed to be running only once across all instances as otherwise every task scheduled by Celerybeat will be submitted multiple times (once for every EC2 instance in the environment). 但是,我希望在我的环境中运行多个实例,因为每个实例都会崩溃,并且在实例重新启动之前需要花费很多时间,但是我无法将当前的体系结构扩展到多个实例,因为Celerybeat应该只在所有实例中运行一次,否则Celerybeat安排的每个任务将被多次提交(对于环境中的每个EC2实例一次)。
I have read about multiple solutions, but all of them seem to have issues that don't make it work for me: 我已经阅读了多个解决方案,但是所有这些解决方案似乎都存在一些问题,这些问题并不适用于我:
How to I use Celery with scheduled tasks in a distributed Elastic Beanstalk environment without task duplication? 如何在分布式Elastic Beanstalk环境中将Celery与计划任务一起使用而不进行任务复制? Eg how can I make sure that exactly one instance is running across all instances all the time in the Elastic Beanstalk environment (even if the current instance with Celerybeat crashes)? 例如,如何确保在Elastic Beanstalk环境中始终在所有实例中运行一个实例(即使Celerybeat的当前实例崩溃)?
Are there any other ways to achieve this? 有没有其他方法可以实现这一目标? What's the best way to use Elastic Beanstalk's Worker Tier Environment with Django? 在Django中使用Elastic Beanstalk的Worker Tier环境的最佳方法是什么?
I guess you could single out celery beat to different group. 我猜你可以把芹菜殴打给不同的团体。
Your auto scaling group runs multiple django instances, but celery is not included in the ec2 config of the scaling group. 您的自动缩放组运行多个django实例,但芹菜不包含在缩放组的ec2配置中。
You should have different set (or just one) of instance for celery beat 你应该有不同的芹菜节拍(或只是一个)实例
In case someone experience similar issues: I ended up switching to a different Queue / Task framework for django. 如果有人遇到类似问题:我最终切换到django的不同队列/任务框架。 It is called django-q and was set up and working in less than an hour. 它被称为django-q,并在不到一个小时的时间内完成了工作。 It has all the features that I needed and also better Django integration than Celery (since djcelery is no longer active). 它具有我需要的所有功能,并且比Celery更好的Django集成(因为djcelery不再活跃)。
Django-q is super easy to use and also lighter than the huge Celery framework. Django-q非常易于使用,也比巨大的Celery框架更轻巧。 I can only recommend it! 我只能推荐它!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.