[英]How do you unit test a Celery task?
The Celery documentation mentions testing Celery within Django but doesn't explain how to test a Celery task if you are not using Django. Celery 文档提到在 Django 中测试 Celery,但没有说明如果您不使用 Django,如何测试 Celery 任务。 How do you do this?
你怎么做到这一点?
It is possible to test tasks synchronously using any unittest lib out there.可以使用任何 unittest 库同步测试任务。 I normaly do 2 different test sessions when working with celery tasks.
在处理 celery 任务时,我通常会进行 2 次不同的测试。 The first one (as I'm suggesting bellow) is completely synchronous and should be the one that makes sure the algorithm does what it should do.
第一个(正如我在下面建议的那样)是完全同步的,应该是确保算法完成它应该做的事情的那个。 The second session uses the whole system (including the broker) and makes sure I'm not having serialization issues or any other distribution, comunication problem.
第二个会话使用整个系统(包括代理)并确保我没有序列化问题或任何其他分发、通信问题。
So:所以:
from celery import Celery
celery = Celery()
@celery.task
def add(x, y):
return x + y
And your test:你的测试:
from nose.tools import eq_
def test_add_task():
rst = add.apply(args=(4, 4)).get()
eq_(rst, 8)
Hope that helps!希望有帮助!
An update to my seven years old answer:更新我七岁的答案:
You can run a worker in a seperate thread via an pytest fixture:您可以通过 pytest 夹具在单独的线程中运行工作人员:
https://docs.celeryq.dev/en/v5.2.6/userguide/testing.html#celery-worker-embed-live-worker https://docs.celeryq.dev/en/v5.2.6/userguide/testing.html#celery-worker-embed-live-worker
According to the docs you should not use "always_eager" (see top of the page of above link).根据文档,您不应使用“always_eager”(请参阅上述链接的页面顶部)。
Old answer:老答案:
I use this:我用这个:
with mock.patch('celeryconfig.CELERY_ALWAYS_EAGER', True, create=True):
...
Docs: https://docs.celeryq.dev/en/3.1/configuration.html#celery-always-eager文档: https ://docs.celeryq.dev/en/3.1/configuration.html#celery-always-eager
CELERY_ALWAYS_EAGER lets you run your task synchronous, and you don't need a celery server. CELERY_ALWAYS_EAGER 让您可以同步运行任务,并且不需要 celery 服务器。
Depends on what exactly you want to be testing.取决于你到底想测试什么。
import unittest
from myproject.myapp import celeryapp
class TestMyCeleryWorker(unittest.TestCase):
def setUp(self):
celeryapp.conf.update(CELERY_ALWAYS_EAGER=True)
# conftest.py
from myproject.myapp import celeryapp
@pytest.fixture(scope='module')
def celery_app(request):
celeryapp.conf.update(CELERY_ALWAYS_EAGER=True)
return celeryapp
# test_tasks.py
def test_some_task(celery_app):
...
from celery import current_app
def send_task(name, args=(), kwargs={}, **opts):
# https://github.com/celery/celery/issues/581
task = current_app.tasks[name]
return task.apply(args, kwargs, **opts)
current_app.send_task = send_task
For those on Celery 4 it's:对于 Celery 4 上的用户,它是:
@override_settings(CELERY_TASK_ALWAYS_EAGER=True)
Because the settings names have been changed and need updating if you choose to upgrade, see因为设置名称已更改,如果您选择升级需要更新,请参阅
https://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html?highlight=what%20is%20new#lowercase-setting-names https://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html?highlight=what%20is%20new#lowercase-setting-names
As of Celery 3.0 , one way to set CELERY_ALWAYS_EAGER
in Django is:从Celery 3.0开始,在Django中设置
CELERY_ALWAYS_EAGER
的一种方法是:
from django.test import TestCase, override_settings
from .foo import foo_celery_task
class MyTest(TestCase):
@override_settings(CELERY_ALWAYS_EAGER=True)
def test_foo(self):
self.assertTrue(foo_celery_task.delay())
Since Celery v4.0 , py.test fixtures are provided to start a celery worker just for the test and are shut down when done:从 Celery v4.0开始, 提供了 py.test 固定装置来启动一个 celery worker 来进行测试,并在完成后关闭:
def test_myfunc_is_executed(celery_session_worker):
# celery_session_worker: <Worker: gen93553@mymachine.local (running)>
assert myfunc.delay().wait(3)
Among other fixtures described on http://docs.celeryproject.org/en/latest/userguide/testing.html#py-test , you can change the celery default options by redefining the celery_config
fixture this way:在http://docs.celeryproject.org/en/latest/userguide/testing.html#py-test上描述的其他固定装置中,您可以通过重新定义
celery_config
固定装置来更改 celery 默认选项:
@pytest.fixture(scope='session')
def celery_config():
return {
'accept_content': ['json', 'pickle'],
'result_serializer': 'pickle',
}
By default, the test worker uses an in-memory broker and result backend.默认情况下,测试工作者使用内存中的代理和结果后端。 No need to use a local Redis or RabbitMQ if not testing specific features.
如果不测试特定功能,则无需使用本地 Redis 或 RabbitMQ。
reference using pytest.使用 pytest参考。
def test_add(celery_worker):
mytask.delay()
if you use flask, set the app config如果您使用烧瓶,请设置应用程序配置
CELERY_BROKER_URL = 'memory://'
CELERY_RESULT_BACKEND = 'cache+memory://'
and in conftest.py
并在
conftest.py
@pytest.fixture
def app():
yield app # Your actual Flask application
@pytest.fixture
def celery_app(app):
from celery.contrib.testing import tasks # need it
yield celery_app # Your actual Flask-Celery application
In my case (and I assume many others), all I wanted was to test the inner logic of a task using pytest.就我而言(我假设还有很多其他人),我想要的只是使用 pytest 测试任务的内部逻辑。
TL;DR; TL;博士; ended up mocking everything away ( OPTION 2 )
最终嘲笑一切(选项2 )
Example Use Case :示例用例:
proj/tasks.py
@shared_task(bind=True)
def add_task(self, a, b):
return a+b;
tests/test_tasks.py
from proj import add_task
def test_add():
assert add_task(1, 2) == 3, '1 + 2 should equal 3'
but, since shared_task
decorator does a lot of celery internal logic, it isn't really a unit tests.但是,由于
shared_task
装饰器做了很多 celery 内部逻辑,它并不是真正的单元测试。
So, for me, there were 2 options:所以,对我来说,有两个选择:
OPTION 1: Separate internal logic选项 1:独立的内部逻辑
proj/tasks_logic.py
def internal_add(a, b):
return a + b;
proj/tasks.py
from .tasks_logic import internal_add
@shared_task(bind=True)
def add_task(self, a, b):
return internal_add(a, b);
This looks very odd, and other than making it less readable, it requires to manually extract and pass attributes that are part of the request, for instance the task_id
in case you need it, which make the logic less pure.这看起来很奇怪,除了使其可读性降低之外,它还需要手动提取和传递作为请求的一部分的属性,例如
task_id
以备不时之需,这使得逻辑不那么纯粹。
OPTION 2: mocks选项 2:模拟
mocking away celery internals嘲笑芹菜内部
tests/__init__.py
# noinspection PyUnresolvedReferences
from celery import shared_task
from mock import patch
def mock_signature(**kwargs):
return {}
def mocked_shared_task(*decorator_args, **decorator_kwargs):
def mocked_shared_decorator(func):
func.signature = func.si = func.s = mock_signature
return func
return mocked_shared_decorator
patch('celery.shared_task', mocked_shared_task).start()
which then allows me to mock the request object (again, in case you need things from the request, like the id, or the retries counter.然后允许我模拟请求对象(同样,如果您需要请求中的内容,例如 id 或重试计数器。
tests/test_tasks.py
from proj import add_task
class MockedRequest:
def __init__(self, id=None):
self.id = id or 1
class MockedTask:
def __init__(self, id=None):
self.request = MockedRequest(id=id)
def test_add():
mocked_task = MockedTask(id=3)
assert add_task(mocked_task, 1, 2) == 3, '1 + 2 should equal 3'
This solution is much more manual, but, it gives me the control I need to actually unit test, without repeating myself, and without losing the celery scope.该解决方案更加手动,但是,它为我提供了实际单元测试所需的控制权,无需重复自己,也不会丢失 celery 范围。
I see a lot of CELERY_ALWAYS_EAGER = true
in unit tests methods as a solution for unit tests, but since the version 5.0.5 is available there are a lot of changes which makes most of the old answers deprecated and for me a time consuming nonsense, so for everyone here searching a Solution, go to the Doc and read the well documented unit test examples for the new Version:我在单元测试方法中看到很多
CELERY_ALWAYS_EAGER = true
作为单元测试的解决方案,但是由于版本 5.0.5 可用,因此有很多更改使得大多数旧答案被弃用,对我来说是一个耗时的废话,因此,对于在这里搜索解决方案的每个人,请转到文档并阅读新版本的有据可查的单元测试示例:
https://docs.celeryproject.org/en/stable/userguide/testing.html https://docs.celeryproject.org/en/stable/userguide/testing.html
And to the Eager Mode with Unit Tests, here a quote from the actual docs:对于带有单元测试的 Eager Mode,这里引用了实际文档的引用:
Eager mode
渴望模式
The eager mode enabled by the task_always_eager setting is by definition not suitable for unit tests.
task_always_eager 设置启用的渴望模式根据定义不适合单元测试。
When testing with eager mode you are only testing an emulation of what happens in a worker, and there are many discrepancies between the emulation and what happens in reality.
在使用 Eager 模式进行测试时,您只是在测试工作人员中发生的情况的仿真,并且仿真与实际发生的情况之间存在许多差异。
Another option is to mock the task if you do not need the side effects of running it.如果您不需要运行任务的副作用,另一种选择是模拟任务。
from unittest import mock
@mock.patch('module.module.task')
def test_name(self, mock_task): ...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.