MongoDB进入Celery任务-Flask应用程序

Question

我正在尝试在Flask应用程序上使用Celery。 我正在文件insight_tasks.py定义任务 。 在该文件中定义了一个函数：

@celery_app.task
def save_insights_task()

该函数做了一些工作，出现了错误，我试图将数据保存到MongoDB中，并且控制台抛出了我： MongoEngineConnectionError（'您尚未定义默认连接'，）所以我认为这是因为MongoEngine尚未初始化，这是我的问题：

我应该如何使用MongoDB的芹菜任务里面？因为我的路线使用的MongoDB（瓶的应用程序）时，它按预期工作。

Celery不共享数据库实例？

档案：

__init __。py（Celery初始化）

celery_app = Celery('insights',
                broker=config.CELERY_LOCATIONS_BROKER_URL,
                backend=config.CELERY_LOCATIONS_RESULT_BACKEND,
                include=['app.insight_tasks']
                )

sight_tasks.py

from app.google import google_service
from app.models import LocationStats
from . import celery_app
from firebase_admin import db as firebase_db
import arrow


@celery_app.task
def save_insight_task(account_location, uid, gid, locations_obj, aggregation):

    try:
        insights, reviews = google_service.store_location_resources(
            gid, uid,
            start_datetime, end_datetime,
            account_location, aggregation
        )
    except StandardError as err:
        from pprint import pprint
        import traceback
        pprint(err)
        pprint(traceback.print_exc())

    path = 'saved_locations/{}/accounts/{}'.format(gid, account_location)
    location = [loc for loc in locations_obj if loc['name'] == 'accounts/' + account_location]
    if len(location) > 0:
        firebase_db.reference(path).update(location[0])

google_service.store_location_resources()是将数据保存到MongoDB中的函数。 此功能是通过我的应用程序的路由在另一侧使用的，因此它可以按预期工作，但芹菜任务除外

---------

Celery任务被调用到POST请求中

account / routes.py

@account.route('/save/queue', methods=['POST'])
def save_all_locations():
    data = request.data
    dataDict = json.loads(data)
    uid = request.headers.get('uid', None)
    gid = request.headers.get('gid', None)

    account_locations = dataDict['locations']
    locations_obj = dataDict['locations_obj']
    for path in account_locations:
        save_insight_task.delay(account_location=path, uid=uid, gid=gid, locations_obj=locations_obj, aggregate='SOME_TEXT')

Answer 1

您应该连接到任务内部的数据库。 原因是因为子进程（由Celery创建）必须具有自己的mongo客户实例。

此处有更多详细信息：将PyMongo与多处理一起使用

例如，定义utils.py ：

from pymodm import connect
def mongo_connect():
    return connect("mongodb://{0}:{1}/{2}".format(MONGODB['host'],
                                    MONGODB['port'],
                                    MONGODB['db_name']),
                                    alias=MONGODB['db_name'])

然后在Insight_tasks.py中

from utils import mongo_connect
@celery_app.task
def save_insight_task(account_location, uid, gid, locations_obj, aggregation):
    # connect to mongodb
    mongo_connect()
    # do your db operations
    try:
        insights, reviews = google_service.store_location_resources(
            gid, uid,
            start_datetime, end_datetime,
            account_location, aggregation
        )
    except StandardError as err:
        from pprint import pprint
        import traceback

        pprint(err)
        pprint(traceback.print_exc())

    path = 'saved_locations/{}/accounts/{}'.format(gid, account_location)
    location = [loc for loc in locations_obj if loc['name'] == 'accounts/' + account_location]
    if len(location) > 0:
        firebase_db.reference(path).update(location[0])

请注意，我将pymodm软件包而不是mongoengine软件包用作mongo的ODM。