堆棧內存溢出
  簡體   English   中英

如何使用 celery 中的鏈式任務和一個入口點 function 來運行我的所有任務？

[英]How do I use chained tasks in celery with one entrypoint function to run all my tasks?

 原文  2021-05-06 02:51:15  4   1    python/ celery

問題描述

可以說我有以下代碼，我需要依賴於其他任務的任務，但有些任務運行時間很長，並且需要某種.join asyncio 類似的，它等待一個任務結束才能繼續下一個任務，但是我該怎么做beat_schedule中的時間表？

beat_schedule = {
    'every-2-seconds': {
        'task': 'celery_app.tasks.add',
        'schedule': timedelta(seconds=2),
        'args': (5, 8)
    },
    'specified-time': {
        'task': 'celery_app.tasks.add',
        'schedule': crontab(hour=8, minute=50),
        'args': (50, 50)
    }

}

def chain_demo():
    tasks = [
        add_demo.si(10, 7),
        mul_demo.si(10),
        insert_db_demo.si(),
    ]

    chain(*tasks).apply_async()

@app.task
def add_demo(x, y):
    time.sleep(3)
    return x + y


@app.task
def mul_demo(x, y):
    time.sleep(3)
    return x * y


@app.task(ignore_result=True)
def insert_db_demo(result):
    print('insert db , result {}'.format(result))

1 個解決方案

解決方案1
 1  2021-05-06 07:39:28

  1. 使用 celery.chain( chain_demo()的入口點 function celery.chain()應該與任務簽名（使用 .s() ）一起傳遞，這些任務簽名（使用.s() ）彼此相鄰，例如 chain(add.s(4) | chain(add.s(4) | mul.s(8)) 。 參考： https://docs.celeryproject.org/en/latest/getting-started/next-steps.html#chains

2.1。 If you want to use celery_beat to perform periodic/scheduled calls to the entrypoint function chain_demo() , then you must make it a celery task instead of an ordinary python function. 這就是beat_schedule配置要求任務名稱的原因。 如https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html中所述

celery beat是一個調度器； 它定期啟動任務，...

2.2. 如果您有其他事情要調用入口點 function chain_demo()而不是使用 celery_beat 例如，如果您要從 Django 應用程序調用它，或者您將從普通 Z23EEEB4347BDD26BFC6B 調用它，那么它不一定需要 5DD'DAZB7是一個 celery 任務（雖然它也是可能的），它可能只是一個普通的 python function。

./tasks.py

import time

from celery import Celery, chain

app = Celery("tasks")

app.conf.update(
    beat_schedule={
        "every-30-seconds": {
            "task": "tasks.chain_demo",
            "schedule": 30,
        },
    },
)


@app.task
def chain_demo():
    tasks = \
        add_demo.s(10, 7) \
        | mul_demo.s(10) \
        | insert_db_demo.s()

    chain(tasks).apply_async()


def chain_demo_non_task():
    tasks = \
        add_demo.s(10, 7) \
        | mul_demo.s(10) \
        | insert_db_demo.s()

    chain(tasks).apply_async()


@app.task
def add_demo(x, y):
    time.sleep(3)
    return x + y


@app.task
def mul_demo(x, y):
    time.sleep(3)
    return x * y


@app.task(ignore_result=True)
def insert_db_demo(result):
    print('insert db , result {}'.format(result))

運行工人

celery --app=tasks worker --loglevel=INFO

調用 #1：使用 celery_beat

celery --app=tasks beat --loglevel=INFO

結果（每 30 秒調用一次任務）：

# Run 1
[2021-05-06 15:10:37,590: INFO/MainProcess] Received task: tasks.chain_demo[c452cf5c-b7cb-4b2b-9284-ada0b19fdedc]  
[2021-05-06 15:10:37,618: INFO/ForkPoolWorker-2] Task tasks.chain_demo[c452cf5c-b7cb-4b2b-9284-ada0b19fdedc] succeeded in 0.02621765899948514s: None
[2021-05-06 15:10:37,619: INFO/MainProcess] Received task: tasks.add_demo[bfaf0b5a-9cd9-43e6-96b1-9bbf11da1a82]  
[2021-05-06 15:10:40,625: INFO/ForkPoolWorker-2] Task tasks.add_demo[bfaf0b5a-9cd9-43e6-96b1-9bbf11da1a82] succeeded in 3.0041544650002834s: 17
[2021-05-06 15:10:40,627: INFO/MainProcess] Received task: tasks.mul_demo[a104c1e3-175e-4fca-9911-88c30157e96f]  
[2021-05-06 15:10:43,631: INFO/ForkPoolWorker-2] Task tasks.mul_demo[a104c1e3-175e-4fca-9911-88c30157e96f] succeeded in 3.0032322850001947s: 170
[2021-05-06 15:10:43,632: INFO/MainProcess] Received task: tasks.insert_db_demo[fce0d2c4-43cc-452e-b3a7-6f0c39cff9fb]  
[2021-05-06 15:10:43,633: WARNING/ForkPoolWorker-2] insert db , result 170
[2021-05-06 15:10:43,633: INFO/ForkPoolWorker-2] Task tasks.insert_db_demo[fce0d2c4-43cc-452e-b3a7-6f0c39cff9fb] succeeded in 0.00018388900025456678s: None
# Run 2
[2021-05-06 15:11:07,567: INFO/MainProcess] Received task: tasks.chain_demo[f3851f64-24f0-454c-a1db-fe2c6f1c59ce]  
[2021-05-06 15:11:07,569: INFO/ForkPoolWorker-2] Task tasks.chain_demo[f3851f64-24f0-454c-a1db-fe2c6f1c59ce] succeeded in 0.0011563549996935762s: None
[2021-05-06 15:11:07,571: INFO/MainProcess] Received task: tasks.add_demo[0ddf340e-78d5-4b13-9190-6815b1b03106]  
[2021-05-06 15:11:10,575: INFO/ForkPoolWorker-2] Task tasks.add_demo[0ddf340e-78d5-4b13-9190-6815b1b03106] succeeded in 3.0036702569996123s: 17
[2021-05-06 15:11:10,576: INFO/MainProcess] Received task: tasks.mul_demo[e68b81a4-723f-40df-97a0-b3ab63f29e03]  
[2021-05-06 15:11:13,584: INFO/ForkPoolWorker-2] Task tasks.mul_demo[e68b81a4-723f-40df-97a0-b3ab63f29e03] succeeded in 3.007046304999676s: 170
[2021-05-06 15:11:13,588: INFO/MainProcess] Received task: tasks.insert_db_demo[f7d0d9e1-7c2e-48af-a4a8-3d6625d351d3]  
[2021-05-06 15:11:13,589: WARNING/ForkPoolWorker-2] insert db , result 170
[2021-05-06 15:11:13,589: INFO/ForkPoolWorker-2] Task tasks.insert_db_demo[f7d0d9e1-7c2e-48af-a4a8-3d6625d351d3] succeeded in 0.0001733720000629546s: None

調用 #2：使用 python 腳本

>>> import tasks
>>> tasks.chain_demo.apply_async()  # Using the celery task asynchronously
<AsyncResult: 46fe65b2-9575-477e-bcdf-11f223db765b>
>>> tasks.chain_demo()  # Using the celery task directly
>>> tasks.chain_demo_non_task()  # Using the ordinary function
>>>

結果：

# Using the celery task asynchronously
[2021-05-06 15:33:51,209: INFO/MainProcess] Received task: tasks.chain_demo[46fe65b2-9575-477e-bcdf-11f223db765b]  
[2021-05-06 15:33:51,211: INFO/ForkPoolWorker-2] Task tasks.chain_demo[46fe65b2-9575-477e-bcdf-11f223db765b] succeeded in 0.0017580129997440963s: None
[2021-05-06 15:33:51,214: INFO/MainProcess] Received task: tasks.add_demo[9b4e6577-856b-457b-9310-44d7dda8979b]  
[2021-05-06 15:33:54,221: INFO/ForkPoolWorker-2] Task tasks.add_demo[9b4e6577-856b-457b-9310-44d7dda8979b] succeeded in 3.0058788619999177s: 17
[2021-05-06 15:33:54,225: INFO/MainProcess] Received task: tasks.mul_demo[e5f9b57b-290d-4a53-92d2-3a7c69c8890c]  
[2021-05-06 15:33:57,232: INFO/ForkPoolWorker-2] Task tasks.mul_demo[e5f9b57b-290d-4a53-92d2-3a7c69c8890c] succeeded in 3.00442869500057s: 170
[2021-05-06 15:33:57,236: INFO/MainProcess] Received task: tasks.insert_db_demo[28ab6ab3-2401-45be-ab67-bff9febbe856]  
[2021-05-06 15:33:57,239: WARNING/ForkPoolWorker-2] insert db , result 170
[2021-05-06 15:33:57,239: INFO/ForkPoolWorker-2] Task tasks.insert_db_demo[28ab6ab3-2401-45be-ab67-bff9febbe856] succeeded in 0.00012745499952870887s: None
# Using the celery task directly
[2021-05-06 15:34:24,018: INFO/MainProcess] Received task: tasks.add_demo[7395866f-54fe-4d02-8114-ae6ac044e25a]  
[2021-05-06 15:34:27,026: INFO/ForkPoolWorker-2] Task tasks.add_demo[7395866f-54fe-4d02-8114-ae6ac044e25a] succeeded in 3.0071180750001076s: 17
[2021-05-06 15:34:27,030: INFO/MainProcess] Received task: tasks.mul_demo[2a2b1452-51d4-49bd-af06-9248895ecc9f]  
[2021-05-06 15:34:30,039: INFO/ForkPoolWorker-2] Task tasks.mul_demo[2a2b1452-51d4-49bd-af06-9248895ecc9f] succeeded in 3.005947522000497s: 170
[2021-05-06 15:34:30,041: INFO/MainProcess] Received task: tasks.insert_db_demo[7095f69f-a02d-4eb7-815e-22e69e2eb4dd]  
[2021-05-06 15:34:30,041: WARNING/ForkPoolWorker-2] insert db , result 170
[2021-05-06 15:34:30,042: INFO/ForkPoolWorker-2] Task tasks.insert_db_demo[7095f69f-a02d-4eb7-815e-22e69e2eb4dd] succeeded in 0.00013701599982596235s: None
# Using the ordinary function
[2021-05-06 15:34:38,889: INFO/MainProcess] Received task: tasks.add_demo[c0f30ae6-0c92-4991-859a-b44f8ef8f88d]  
[2021-05-06 15:34:41,897: INFO/ForkPoolWorker-2] Task tasks.add_demo[c0f30ae6-0c92-4991-859a-b44f8ef8f88d] succeeded in 3.0069600609995177s: 17
[2021-05-06 15:34:41,899: INFO/MainProcess] Received task: tasks.mul_demo[ec4b1a9f-b8c8-440f-af37-d1a905ce9747]  
[2021-05-06 15:34:44,905: INFO/ForkPoolWorker-2] Task tasks.mul_demo[ec4b1a9f-b8c8-440f-af37-d1a905ce9747] succeeded in 3.0045316010000533s: 170
[2021-05-06 15:34:44,907: INFO/MainProcess] Received task: tasks.insert_db_demo[7d36777b-e8e1-4983-894f-0539f476874d]  
[2021-05-06 15:34:44,908: WARNING/ForkPoolWorker-2] insert db , result 170
[2021-05-06 15:34:44,908: INFO/ForkPoolWorker-2] Task tasks.insert_db_demo[7d36777b-e8e1-4983-894f-0539f476874d] succeeded in 0.00018011399970419006s: None

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 如何使用Celery運行並行任務？ 如何讓多個芹菜工人執行相同的任務？ 如何在 celery 中將任務從一個隊列移動到另一個隊列 如何覆蓋 celery 任務的后端 是否可以確保已鏈接的celery任務在同一節點worker上運行？ Celery 如何用 Python 運行多個任務這么慢？ 在繼續我的代碼之前，我如何等到所有 celery 任務完成 Python線程-如何使用它一次運行多個任務？ 如何使用Celery制作包含所有待處理任務的儀表板？ 如何在運行任務之前設置Celery來調用自定義初始化函數？
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM