Celery beat stuck on starting with Django and Redis

Question

In my Django project I have two tasks in two different apps that I want to run periodically with Celery.

The worker seems to collect the tasks and the beat seems to collect the schedular. However, the beat stuck on starting (it didn't synchronize the schedules) and never deliver the tasks to the worker.

The celery --app=bozonaro worker --loglevel=debug --beat ( bozonaro is my django project's name) command prompts the following to me:

[2021-02-04 18:23:48,080: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2021-02-04 18:23:48,103: DEBUG/MainProcess] | Worker: Building graph...
[2021-02-04 18:23:48,104: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2021-02-04 18:23:48,257: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2021-02-04 18:23:48,257: DEBUG/MainProcess] | Consumer: Building graph...
[2021-02-04 18:23:48,413: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Agent, Mingle, Gossip, Tasks, Control, event loop}
 
 -------------- celery@LAPTOP-E5L3SQ6N v5.0.5 (singularity)
--- ***** ----- 
-- ******* ---- Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.29 2021-02-04 18:23:48
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         bozonaro:0x7fc00b16d4c0
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . actions.tasks.post_action_tweet
  . bozonaro.celery.debug_task
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . quotes.tasks.post_quote_tweet

[2021-02-04 18:23:48,468: DEBUG/MainProcess] | Worker: Starting Hub
[2021-02-04 18:23:48,468: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,468: DEBUG/MainProcess] | Worker: Starting Pool
[2021-02-04 18:23:48,538: DEBUG/ForkPoolWorker-2] Using selector: EpollSelector
[2021-02-04 18:23:48,574: DEBUG/ForkPoolWorker-3] Using selector: EpollSelector
[2021-02-04 18:23:48,608: DEBUG/ForkPoolWorker-4] Using selector: EpollSelector
[2021-02-04 18:23:48,641: DEBUG/ForkPoolWorker-5] Using selector: EpollSelector
[2021-02-04 18:23:48,675: DEBUG/ForkPoolWorker-6] Using selector: EpollSelector
[2021-02-04 18:23:48,708: DEBUG/ForkPoolWorker-7] Using selector: EpollSelector
[2021-02-04 18:23:48,741: DEBUG/ForkPoolWorker-8] Using selector: EpollSelector
[2021-02-04 18:23:48,773: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,773: DEBUG/MainProcess] | Worker: Starting Beat
[2021-02-04 18:23:48,774: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,775: DEBUG/MainProcess] | Worker: Starting Consumer
[2021-02-04 18:23:48,775: DEBUG/MainProcess] | Consumer: Starting Connection
[2021-02-04 18:23:48,776: DEBUG/ForkPoolWorker-9] Using selector: EpollSelector
[2021-02-04 18:23:48,781: INFO/Beat] beat: Starting...
[2021-02-04 18:23:48,784: INFO/MainProcess] Connected to redis://localhost:6379//
[2021-02-04 18:23:48,784: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,784: DEBUG/MainProcess] | Consumer: Starting Events
[2021-02-04 18:23:48,790: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,790: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-02-04 18:23:48,792: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,792: DEBUG/MainProcess] | Consumer: Starting Mingle
[2021-02-04 18:23:48,792: INFO/MainProcess] mingle: searching for neighbors
[2021-02-04 18:23:48,810: DEBUG/Beat] Current schedule:
<ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)>
<ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>
[2021-02-04 18:23:48,810: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes
[2021-02-04 18:23:48,811: DEBUG/Beat] beat: Waking up in 5.00 minutes.
[2021-02-04 18:23:49,808: INFO/MainProcess] mingle: all alone
[2021-02-04 18:23:49,809: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,809: DEBUG/MainProcess] | Consumer: Starting Gossip
[2021-02-04 18:23:49,813: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,813: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-02-04 18:23:49,831: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,831: DEBUG/MainProcess] | Consumer: Starting Control
[2021-02-04 18:23:49,833: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,833: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-02-04 18:23:49,833: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2021-02-04 18:23:49,833: INFO/MainProcess] celery@LAPTOP-E5L3SQ6N ready.
[2021-02-04 18:23:49,833: DEBUG/MainProcess] basic.qos: prefetch_count->32
[2021-02-04 18:28:48,850: DEBUG/Beat] beat: Synchronizing schedule...
[2021-02-04 18:28:48,856: DEBUG/Beat] beat: Waking up in 5.00 minutes.
[2021-02-04 18:33:48,909: DEBUG/Beat] beat: Synchronizing schedule...
[2021-02-04 18:33:48,918: DEBUG/Beat] beat: Waking up in 5.00 minutes.

As you can see the beat are failing to synchronize the schedules, which causes an impossibility to write the entries and so on...

I've lost a lot of time looking for this solution but everything that I found didn't work or was from an older version of Celery. So, what am I'm missing?

The rest of my configurations:

bozonaro/ init .py

from .celery import app as celery_app

__all__ = ("celery_app",)

bozonaro/settings.py

from decouple import config as secret


CELERY_BROKER_URL = os.environ.get("REDIS_URL", secret("REDIS_URL"))
CELERY_RESULT_BACKEND = os.environ.get("REDIS_URL", secret("REDIS_URL"))
CELERY_ACCEPT_CONTENT = ["application/json"]
CELERY_RESULT_SERIALIZER = "json"
CELERY_TASK_SERIALIZER = "json"

bozonaro/celery.py

import os

from celery import Celery
from celery.schedules import crontab

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "bozonaro.settings")

app = Celery("bozonaro")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()


app.conf.beat_schedule = {
    "post-action-tweet-every-day-at-8h": {
        "task": "actions.tasks.post_action_tweet",
        "schedule": crontab(minute=0, hour=8),
    },
    "post-quote-tweet-every-day-at-18h": {
        "task": "quotes.tasks.post_quote_tweet",
        "schedule": crontab(minute=0, hour=18),
    },
}


@app.task(bind=True)
def debug_task(self):
    print(f"Request: {self.request!r}")

actions/tasks.py

from celery import shared_task

from core.tasks_helper import get_generic_request, post_entity_tweet
from .views import get_random_action


@shared_task
def post_action_tweet() -> None:
    action = get_random_action(request=get_generic_request()).data
    post_entity_tweet(infos=action, type_="action")

quotes/tasks.py

from celery import shared_task

from core.tasks_helper import get_generic_request, post_entity_tweet
from .views import get_random_quote


@shared_task
def post_quote_tweet() -> None:
    quote = get_random_quote(request=get_generic_request()).data
    post_entity_tweet(infos=quote, type_="quote")

I'm using Python 3.9.1, Ubuntu 20.04 (on WSL), Celery 5.0.5 and Redis 3.5.3.

Thanks a lot!

EDIT:

• " what it is that isn't working? "

  • The beat is not sending the tasks to the worker.

• " When you say that celery-beat is failing to synchronize the schedules, what do you mean by that? "

  • It is just an interpretation from the logs. As you can see in the last lines of it, the beat will endlessly repeat the following pattern:

     beat: Synchronizing schedule... beat: Waking up in 5.00 minutes. beat: Synchronizing schedule... beat: Waking up in 5.00 minutes.

• " Also, please explain what you expect to happen versus what is actually happening -- or failing to happen. "

  • I expect the beat to send the tasks to the worker at the specific time. The Schedule seems to be right...

     Current schedule: <ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)> <ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>

  • ...but nothing happens in the worker.

Answer 1

The celery beat scheduler seems to be working as expected from your current configuration.

Your tasks are scheduled to run at 08:00 daily and 18:00 daily based on your configuration:

app.conf.beat_schedule = {
    "post-action-tweet-every-day-at-8h": {
        "task": "actions.tasks.post_action_tweet",
        "schedule": crontab(minute=0, hour=8),      # 08:00 daily
    },
    "post-quote-tweet-every-day-at-18h": {
        "task": "quotes.tasks.post_quote_tweet",
        "schedule": crontab(minute=0, hour=18),     # 18:00 daily
    },
}

When you start the celery beat scheduler process, it reflects this:

[2021-02-04 18:23:48,810: DEBUG/Beat] Current schedule:
<ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)>
<ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>

The tasks are scheduled to run at 08:00 and 18:00 daily. The time that the scheduler process was run was at [2021-02-04 18:23:48,810: DEBUG/Beat] , which is after both tasks are scheduled to run for the day .

More info for crontab() objects can be found in the celery documentation .

If instead of at 08:00 daily you meant every 8 hours , you would update your configuration to use:

crontab(minute=0, hour="*/8")