[英]How to use Flask-SQLAlchemy in a Celery task
I recently switch to Celery 3.0.我最近切换到 Celery 3.0。 Before that I was using Flask-Celery in order to integrate Celery with Flask.
在此之前,我使用Flask-Celery来将 Celery 与 Flask 集成。 Although it had many issues like hiding some powerful Celery functionalities but it allowed me to use the full context of Flask app and especially Flask-SQLAlchemy.
虽然它有很多问题,比如隐藏了一些强大的 Celery 功能,但它允许我使用 Flask 应用程序的完整上下文,尤其是 Flask-SQLAlchemy。
In my background tasks I am processing data and the SQLAlchemy ORM to store the data.在我的后台任务中,我正在处理数据和 SQLAlchemy ORM 来存储数据。 The maintainer of Flask-Celery has dropped support of the plugin.
Flask-Celery 的维护者已经放弃了对该插件的支持。 The plugin was pickling the Flask instance in the task so I could have full access to SQLAlchemy.
该插件正在处理任务中的 Flask 实例,以便我可以完全访问 SQLAlchemy。
I am trying to replicate this behavior in my tasks.py file but with no success.我试图在我的 tasks.py 文件中复制此行为,但没有成功。 Do you have any hints on how to achieve this?
您对如何实现这一目标有任何提示吗?
extensions.py扩展.py
import flask
from flask.ext.sqlalchemy import SQLAlchemy
from celery import Celery
class FlaskCelery(Celery):
def __init__(self, *args, **kwargs):
super(FlaskCelery, self).__init__(*args, **kwargs)
self.patch_task()
if 'app' in kwargs:
self.init_app(kwargs['app'])
def patch_task(self):
TaskBase = self.Task
_celery = self
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
if flask.has_app_context():
return TaskBase.__call__(self, *args, **kwargs)
else:
with _celery.app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
self.Task = ContextTask
def init_app(self, app):
self.app = app
self.config_from_object(app.config)
celery = FlaskCelery()
db = SQLAlchemy()
app.py应用程序
from flask import Flask
from extensions import celery, db
def create_app():
app = Flask()
#configure/initialize all your extensions
db.init_app(app)
celery.init_app(app)
return app
Once you've set up your app this way, you can run and use celery without having to explicitly run it from within an application context, as all your tasks will automatically be run in an application context if necessary, and you don't have to explicitly worry about post-task teardown, which is an important issue to manage (see other responses below).一旦您以这种方式设置了您的应用程序,您就可以运行和使用 celery,而无需在应用程序上下文中显式运行它,因为您的所有任务将在必要时自动在应用程序上下文中运行,而您没有明确担心任务后拆卸,这是一个需要管理的重要问题(请参阅下面的其他回复)。
Those who keep getting with _celery.app.app_context(): AttributeError: 'FlaskCelery' object has no attribute 'app'
make sure to:那些继续
with _celery.app.app_context(): AttributeError: 'FlaskCelery' object has no attribute 'app'
确保:
celery
import at the app.py
file level.celery
导入保持在app.py
文件级别。 Avoid: app.py应用程序
from flask import Flask
def create_app():
app = Flask()
initiliaze_extensions(app)
return app
def initiliaze_extensions(app):
from extensions import celery, db # DOOMED! Keep celery import at the FILE level
db.init_app(app)
celery.init_app(app)
flask run
and useflask run
和使用之前开始你的芹菜工人celery worker -A app:celery -l info -f celery.log
Note the app:celery
, ie loading from app.py
.注意
app:celery
,即从app.py
加载。
You can still import from extensions to decorate tasks, ie from extensions import celery
.您仍然可以从扩展导入来装饰任务,即
from extensions import celery
。
I prefer to run all of celery within the application context by creating a separate file that invokes celery.start() with the application's context.我更喜欢通过创建一个单独的文件来在应用程序上下文中运行所有 celery,该文件使用应用程序的上下文调用 celery.start()。 This means your tasks file doesn't have to be littered with context setup and teardowns.
这意味着您的任务文件不必充斥着上下文设置和拆卸。 It also lends itself well to the flask 'application factory' pattern.
它还非常适合烧瓶“应用程序工厂”模式。
extensions.py扩展.py
from from flask.ext.sqlalchemy import SQLAlchemy
from celery import Celery
db = SQLAlchemy()
celery = Celery()
tasks.py任务.py
from extensions import celery, db
from flask.globals import current_app
from celery.signals import task_postrun
@celery.task
def do_some_stuff():
current_app.logger.info("I have the application context")
#you can now use the db object from extensions
@task_postrun.connect
def close_session(*args, **kwargs):
# Flask SQLAlchemy will automatically create new sessions for you from
# a scoped session factory, given that we are maintaining the same app
# context, this ensures tasks have a fresh session (e.g. session errors
# won't propagate across tasks)
db.session.remove()
app.py应用程序
from extensions import celery, db
def create_app():
app = Flask()
#configure/initialize all your extensions
db.init_app(app)
celery.config_from_object(app.config)
return app
RunCelery.py运行Celery.py
from app import create_app
from extensions import celery
app = create_app()
if __name__ == '__main__':
with app.app_context():
celery.start()
I used Paul Gibbs' answer with two differences.我使用了Paul Gibbs 的答案,但有两个不同之处。 Instead of task_postrun I used worker_process_init.
我使用了 worker_process_init 而不是 task_postrun。 And instead of .remove() I used db.session.expire_all().
而不是 .remove() 我使用了 db.session.expire_all()。
I'm not 100% sure, but from what I understand the way this works is when Celery creates a worker process, all inherited/shared db sessions will be expired, and SQLAlchemy will create new sessions on demand unique to that worker process.我不是 100% 确定,但从我理解的工作方式来看,当 Celery 创建一个工作进程时,所有继承/共享的数据库会话都将过期,并且 SQLAlchemy 将根据需要创建该工作进程独有的新会话。
So far it seems to have fixed my problem.到目前为止,它似乎已经解决了我的问题。 With Paul's solution, when one worker finished and removed the session, another worker using the same session was still running its query, so db.session.remove() closed the connection while it was being used, giving me a "Lost connection to MySQL server during query" exception.
使用 Paul 的解决方案,当一个 worker 完成并删除会话时,另一个使用相同会话的 worker 仍在运行其查询,因此 db.session.remove() 在使用时关闭了连接,给我一个“与 MySQL 的连接丢失查询期间的服务器”异常。
Thanks Paul for steering me in the right direction!感谢保罗引导我朝着正确的方向前进!
Nevermind that didn't work.没关系那没有用。 I ended up having an argument in my Flask app factory to not run db.init_app(app) if Celery was calling it.
如果 Celery 调用它,我最终在我的 Flask 应用程序工厂中争论不运行 db.init_app(app)。 Instead the workers will call it after Celery forks them.
相反,工作人员会在 Celery 对它们进行分叉后调用它。 I now see several connections in my MySQL processlist.
我现在在我的 MySQL 进程列表中看到了几个连接。
from extensions import db
from celery.signals import worker_process_init
from flask import current_app
@worker_process_init.connect
def celery_worker_init_db(**_):
db.init_app(current_app)
In your tasks.py file do the following:在您的 tasks.py 文件中执行以下操作:
from main import create_app
app = create_app()
celery = Celery(__name__)
celery.add_defaults(lambda: app.config)
@celery.task
def create_facet(project_id, **kwargs):
with app.test_request_context():
# your code
from flask import Flask
from werkzeug.utils import import_string
from celery.signals import worker_process_init, celeryd_init
from flask_celery import Celery
from src.app import config_from_env, create_app
celery = Celery()
def get_celery_conf():
config = import_string('src.settings')
config = {k: getattr(config, k) for k in dir(config) if k.isupper()}
config['BROKER_URL'] = config['CELERY_BROKER_URL']
return config
@celeryd_init.connect
def init_celeryd(conf=None, **kwargs):
conf.update(get_celery_conf())
@worker_process_init.connect
def init_celery_flask_app(**kwargs):
app = create_app()
app.app_context().push()
By doing this, we are able to maintain database connection per-worker.通过这样做,我们能够维护每个工人的数据库连接。
If you want to run your task under flask context, you can subclass Task.__call__
:如果你想在
Task.__call__
上下文下运行你的任务,你可以Task.__call__
:
class SmartTask(Task):
abstract = True
def __call__(self, *_args, **_kwargs):
with self.app.flask_app.app_context():
with self.app.flask_app.test_request_context():
result = super(SmartTask, self).__call__(*_args, **_kwargs)
return result
class SmartCelery(Celery):
def init_app(self, app):
super(SmartCelery, self).init_app(app)
self.Task = SmartTask
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.