MongoDB into a Celery Task - Flask Application

Question

I'm trying to use Celery on my Flask application. I'm defining a task in a file insight_tasks.py . In that file is defined a function:

@celery_app.task
def save_insights_task()

That function do some stuff and, here comes the error, I'm trying to save data into MongoDB and the console throws me: MongoEngineConnectionError('You have not defined a default connection',) So I think it's because MongoEngine has not been initialized, and here is my question:

How should I use MongoDB inside a Celery Task?, because when using MongoDB on my routes (flask app) It's working as expected.

Celery does not share the db instance?

Files:

__init__.py (Celery intialization)

celery_app = Celery('insights',
                broker=config.CELERY_LOCATIONS_BROKER_URL,
                backend=config.CELERY_LOCATIONS_RESULT_BACKEND,
                include=['app.insight_tasks']
                )

insight_tasks.py

from app.google import google_service
from app.models import LocationStats
from . import celery_app
from firebase_admin import db as firebase_db
import arrow


@celery_app.task
def save_insight_task(account_location, uid, gid, locations_obj, aggregation):

    try:
        insights, reviews = google_service.store_location_resources(
            gid, uid,
            start_datetime, end_datetime,
            account_location, aggregation
        )
    except StandardError as err:
        from pprint import pprint
        import traceback
        pprint(err)
        pprint(traceback.print_exc())

    path = 'saved_locations/{}/accounts/{}'.format(gid, account_location)
    location = [loc for loc in locations_obj if loc['name'] == 'accounts/' + account_location]
    if len(location) > 0:
        firebase_db.reference(path).update(location[0])

Here google_service.store_location_resources() is the function that saves thew data into MongoDB. this function is used on another side, by the routes of my app, so it works as expected, except on the Celery task

---------

The Celery task is called into a POST request

accounts/routes.py

@account.route('/save/queue', methods=['POST'])
def save_all_locations():
    data = request.data
    dataDict = json.loads(data)
    uid = request.headers.get('uid', None)
    gid = request.headers.get('gid', None)

    account_locations = dataDict['locations']
    locations_obj = dataDict['locations_obj']
    for path in account_locations:
        save_insight_task.delay(account_location=path, uid=uid, gid=gid, locations_obj=locations_obj, aggregate='SOME_TEXT')

Answer 1

You are supposed to connect to the database inside the task. The reason is because child processes (created by Celery) must have their own instance of mongo client.

More details here : Using PyMongo with Multiprocessing

For example define a utils.py :

from pymodm import connect
def mongo_connect():
    return connect("mongodb://{0}:{1}/{2}".format(MONGODB['host'],
                                    MONGODB['port'],
                                    MONGODB['db_name']),
                                    alias=MONGODB['db_name'])

Then in insight_tasks.py

from utils import mongo_connect
@celery_app.task
def save_insight_task(account_location, uid, gid, locations_obj, aggregation):
    # connect to mongodb
    mongo_connect()
    # do your db operations
    try:
        insights, reviews = google_service.store_location_resources(
            gid, uid,
            start_datetime, end_datetime,
            account_location, aggregation
        )
    except StandardError as err:
        from pprint import pprint
        import traceback

        pprint(err)
        pprint(traceback.print_exc())

    path = 'saved_locations/{}/accounts/{}'.format(gid, account_location)
    location = [loc for loc in locations_obj if loc['name'] == 'accounts/' + account_location]
    if len(location) > 0:
        firebase_db.reference(path).update(location[0])

Note that I use pymodm package instead of mongoengine package as ODM for mongo.