设置芹菜周期性任务

Question

How do I set up a periodic task with Celerybeat and Flask that queries a database every hour?

The environment looks like this:

/
|-app
  |-__init__.py
  |-jobs
    |-task.py
|-celery-beat.sh
|-celery-worker.sh
|-manage.py

I currently have a query function called run_query() located in task.py

I want the scheduler to kick in once the application initiates so I have the following lines in my /app/__init__.py folder:

celery = Celery()

@celery.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
  sender.add_periodic_task(1, app.jobs.task.run_query())

(For simplicity's sake, I've set it up so that if it runs, it will run every minute. No such luck yet.)

When I launch the celery-worker.sh it recognizes my function under the [tasks] heading. But the scheduled function never runs. I can manually force the function to run by issuing the following at the command prompt:

>> from app.jobs import task
>> task.run_query.delay()

EDIT : Added celerybeat.sh

As a follow up: If the database is accessed through a flask context, during my asynch function call is it wise to create a new flask context to access the database? Use the existing flask context? Or forget contexts altogether and just initiate a connection to the database? My worry is that if I just initiate a new connection it may interfere with the existing context's connection?

Answer 1

To run periodic tasks you need some kind of schduler (Eg. celery beat).

celery beat is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster.

You have to ensure only a single scheduler is running for a schedule at a time, otherwise you'd end up with duplicate tasks. Using a centralized approach means the schedule doesn't have to be synchronized, and the service can operate without using locks.

Reference: periodic-tasks

You can invoke scheduler with command,

$ celery -A proj beat #different process from your worker

You can also embed beat inside the worker by enabling the workers -B option, this is convenient if you'll never run more than one worker node, but it's not commonly used and for that reason isn't recommended for production use Starting scheduler:

$ celery -A proj worker -B

Reference: celery starting scheduler