堆栈内存溢出
  简体   繁体   English

Celery 节拍卡在从 Django 和 Redis 开始

[英]Celery beat stuck on starting with Django and Redis

 原文  2021-02-04 19:04:39  0   1    python/ django/ redis/ celery/ celerybeat

问题描述

In my Django project I have two tasks in two different apps that I want to run periodically with Celery.在我的 Django 项目中，我在两个不同的应用程序中有两个任务，我想使用 Celery 定期运行它们。

The worker seems to collect the tasks and the beat seems to collect the schedular.工人似乎在收集任务，节拍似乎在收集时间表。 However, the beat stuck on starting (it didn't synchronize the schedules) and never deliver the tasks to the worker.但是，节拍卡在开始（它没有同步时间表）并且从未将任务交付给工作人员。

The celery --app=bozonaro worker --loglevel=debug --beat ( bozonaro is my django project's name) command prompts the following to me: celery --app=bozonaro worker --loglevel=debug --beat （ bozonaro是我的 django 项目的名称）命令向我提示以下内容：

[2021-02-04 18:23:48,080: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2021-02-04 18:23:48,103: DEBUG/MainProcess] | Worker: Building graph...
[2021-02-04 18:23:48,104: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2021-02-04 18:23:48,257: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2021-02-04 18:23:48,257: DEBUG/MainProcess] | Consumer: Building graph...
[2021-02-04 18:23:48,413: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Agent, Mingle, Gossip, Tasks, Control, event loop}
 
 -------------- celery@LAPTOP-E5L3SQ6N v5.0.5 (singularity)
--- ***** ----- 
-- ******* ---- Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.29 2021-02-04 18:23:48
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         bozonaro:0x7fc00b16d4c0
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . actions.tasks.post_action_tweet
  . bozonaro.celery.debug_task
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . quotes.tasks.post_quote_tweet

[2021-02-04 18:23:48,468: DEBUG/MainProcess] | Worker: Starting Hub
[2021-02-04 18:23:48,468: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,468: DEBUG/MainProcess] | Worker: Starting Pool
[2021-02-04 18:23:48,538: DEBUG/ForkPoolWorker-2] Using selector: EpollSelector
[2021-02-04 18:23:48,574: DEBUG/ForkPoolWorker-3] Using selector: EpollSelector
[2021-02-04 18:23:48,608: DEBUG/ForkPoolWorker-4] Using selector: EpollSelector
[2021-02-04 18:23:48,641: DEBUG/ForkPoolWorker-5] Using selector: EpollSelector
[2021-02-04 18:23:48,675: DEBUG/ForkPoolWorker-6] Using selector: EpollSelector
[2021-02-04 18:23:48,708: DEBUG/ForkPoolWorker-7] Using selector: EpollSelector
[2021-02-04 18:23:48,741: DEBUG/ForkPoolWorker-8] Using selector: EpollSelector
[2021-02-04 18:23:48,773: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,773: DEBUG/MainProcess] | Worker: Starting Beat
[2021-02-04 18:23:48,774: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,775: DEBUG/MainProcess] | Worker: Starting Consumer
[2021-02-04 18:23:48,775: DEBUG/MainProcess] | Consumer: Starting Connection
[2021-02-04 18:23:48,776: DEBUG/ForkPoolWorker-9] Using selector: EpollSelector
[2021-02-04 18:23:48,781: INFO/Beat] beat: Starting...
[2021-02-04 18:23:48,784: INFO/MainProcess] Connected to redis://localhost:6379//
[2021-02-04 18:23:48,784: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,784: DEBUG/MainProcess] | Consumer: Starting Events
[2021-02-04 18:23:48,790: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,790: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-02-04 18:23:48,792: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:48,792: DEBUG/MainProcess] | Consumer: Starting Mingle
[2021-02-04 18:23:48,792: INFO/MainProcess] mingle: searching for neighbors
[2021-02-04 18:23:48,810: DEBUG/Beat] Current schedule:
<ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)>
<ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>
[2021-02-04 18:23:48,810: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes
[2021-02-04 18:23:48,811: DEBUG/Beat] beat: Waking up in 5.00 minutes.
[2021-02-04 18:23:49,808: INFO/MainProcess] mingle: all alone
[2021-02-04 18:23:49,809: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,809: DEBUG/MainProcess] | Consumer: Starting Gossip
[2021-02-04 18:23:49,813: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,813: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-02-04 18:23:49,831: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,831: DEBUG/MainProcess] | Consumer: Starting Control
[2021-02-04 18:23:49,833: DEBUG/MainProcess] ^-- substep ok
[2021-02-04 18:23:49,833: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-02-04 18:23:49,833: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2021-02-04 18:23:49,833: INFO/MainProcess] celery@LAPTOP-E5L3SQ6N ready.
[2021-02-04 18:23:49,833: DEBUG/MainProcess] basic.qos: prefetch_count->32
[2021-02-04 18:28:48,850: DEBUG/Beat] beat: Synchronizing schedule...
[2021-02-04 18:28:48,856: DEBUG/Beat] beat: Waking up in 5.00 minutes.
[2021-02-04 18:33:48,909: DEBUG/Beat] beat: Synchronizing schedule...
[2021-02-04 18:33:48,918: DEBUG/Beat] beat: Waking up in 5.00 minutes.

As you can see the beat are failing to synchronize the schedules, which causes an impossibility to write the entries and so on...如您所见，节拍未能同步时间表，这导致无法写入条目等等......

I've lost a lot of time looking for this solution but everything that I found didn't work or was from an older version of Celery.我已经浪费了很多时间来寻找这个解决方案，但是我发现的所有东西都不起作用或者来自旧版本的 Celery。 So, what am I'm missing?那么，我错过了什么？

The rest of my configurations:我的配置的rest：

bozonaro/ init .py bozonaro/初始化.py

from .celery import app as celery_app

__all__ = ("celery_app",)

bozonaro/settings.py bozonaro/settings.py

from decouple import config as secret


CELERY_BROKER_URL = os.environ.get("REDIS_URL", secret("REDIS_URL"))
CELERY_RESULT_BACKEND = os.environ.get("REDIS_URL", secret("REDIS_URL"))
CELERY_ACCEPT_CONTENT = ["application/json"]
CELERY_RESULT_SERIALIZER = "json"
CELERY_TASK_SERIALIZER = "json"

bozonaro/celery.py bozonaro/celery.py

import os

from celery import Celery
from celery.schedules import crontab

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "bozonaro.settings")

app = Celery("bozonaro")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()


app.conf.beat_schedule = {
    "post-action-tweet-every-day-at-8h": {
        "task": "actions.tasks.post_action_tweet",
        "schedule": crontab(minute=0, hour=8),
    },
    "post-quote-tweet-every-day-at-18h": {
        "task": "quotes.tasks.post_quote_tweet",
        "schedule": crontab(minute=0, hour=18),
    },
}


@app.task(bind=True)
def debug_task(self):
    print(f"Request: {self.request!r}")

actions/tasks.py动作/tasks.py

from celery import shared_task

from core.tasks_helper import get_generic_request, post_entity_tweet
from .views import get_random_action


@shared_task
def post_action_tweet() -> None:
    action = get_random_action(request=get_generic_request()).data
    post_entity_tweet(infos=action, type_="action")

quotes/tasks.py引号/tasks.py

from celery import shared_task

from core.tasks_helper import get_generic_request, post_entity_tweet
from .views import get_random_quote


@shared_task
def post_quote_tweet() -> None:
    quote = get_random_quote(request=get_generic_request()).data
    post_entity_tweet(infos=quote, type_="quote")

I'm using Python 3.9.1, Ubuntu 20.04 (on WSL), Celery 5.0.5 and Redis 3.5.3. I'm using Python 3.9.1, Ubuntu 20.04 (on WSL), Celery 5.0.5 and Redis 3.5.3.

Thanks a lot!非常感谢！

EDIT:编辑：

  • " what it is that isn't working? " “什么不工作？ ”

    • The beat is not sending the tasks to the worker.节拍没有将任务发送给工人。

  • " When you say that celery-beat is failing to synchronize the schedules, what do you mean by that? " “当你说 celery-beat 无法同步时间表时，你是什么意思？ ”

    • It is just an interpretation from the logs.这只是日志的解释。 As you can see in the last lines of it, the beat will endlessly repeat the following pattern:正如您在它的最后几行中看到的那样，节拍将无休止地重复以下模式：

       beat: Synchronizing schedule... beat: Waking up in 5.00 minutes. beat: Synchronizing schedule... beat: Waking up in 5.00 minutes.

  • " Also, please explain what you expect to happen versus what is actually happening -- or failing to happen. " “另外，请解释你期望发生的事情与实际发生的事情——或没有发生的事情。 ”

    • I expect the beat to send the tasks to the worker at the specific time.我希望节拍在特定时间将任务发送给工作人员。 The Schedule seems to be right...时间表似乎是正确的...

       Current schedule: <ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)> <ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>

    • ...but nothing happens in the worker. ...但是工人没有任何反应。

1 个解决方案

解决方案1
 1  2021-02-09 21:33:50

The celery beat scheduler seems to be working as expected from your current configuration. celery 节拍调度程序似乎在您当前的配置中按预期工作。

Your tasks are scheduled to run at 08:00 daily and 18:00 daily based on your configuration:根据您的配置，您的任务计划在每天 08:00 和每天18:00运行：

app.conf.beat_schedule = {
    "post-action-tweet-every-day-at-8h": {
        "task": "actions.tasks.post_action_tweet",
        "schedule": crontab(minute=0, hour=8),      # 08:00 daily
    },
    "post-quote-tweet-every-day-at-18h": {
        "task": "quotes.tasks.post_quote_tweet",
        "schedule": crontab(minute=0, hour=18),     # 18:00 daily
    },
}

When you start the celery beat scheduler process, it reflects this:当您启动 celery 节拍调度程序进程时，它反映了这一点：

[2021-02-04 18:23:48,810: DEBUG/Beat] Current schedule:
<ScheduleEntry: post-quote-tweet-every-day-at-18h quotes.tasks.post_quote_tweet() <crontab: 0 18 * * * (m/h/d/dM/MY)>
<ScheduleEntry: post-action-tweet-every-day-at-8h actions.tasks.post_action_tweet() <crontab: 0 8 * * * (m/h/d/dM/MY)>

The tasks are scheduled to run at 08:00 and 18:00 daily.这些任务计划在每天的 08:00 和 18:00 运行。 The time that the scheduler process was run was at [2021-02-04 18:23:48,810: DEBUG/Beat] , which is after both tasks are scheduled to run for the day .调度程序进程的运行时间是[2021-02-04 18:23:48,810: DEBUG/Beat] ，这是在两个任务都安排在当天运行之后。

More info for crontab() objects can be found in the celery documentation .更多关于crontab()对象的信息可以在celery 文档中找到。

If instead of at 08:00 daily you meant every 8 hours , you would update your configuration to use:如果您的意思不是每天 08:00 ，而是每 8 小时，您将更新配置以使用：

crontab(minute=0, hour="*/8")

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 芹菜击败不是从Heroku开始 - Celery beat not starting on Heroku django芹菜击败DBAccessError - django celery beat DBAccessError django celeryd和芹菜节拍 - django celeryd and celery beat 姜戈芹菜，芹菜打败工人 - Django Celery, Celery beat workers 设置 docker-compose.yml 以运行 celery worker 和 celery beat 以 redis 作为代理的 django 项目 - Setting up docker-compose.yml to run celery worker and celery beat for a django project with redis as broker Django Celery 和 Django-Celery-Beat - Django Celery & Django-Celery-Beat Django芹菜节拍任务不起作用 - Django celery beat task not working 如何启动和停止 celery 和 Django 中的工人？ - How to start and stop celery beat and workers in Django? Celery击败Windows简单示例（不使用Django） - Celery Beat Windows Simple Example (not with Django) Django Celery Beat with Database scheduler not running tasks - Django Celery Beat with Database scheduler not running tasks
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM