Celery 容器在使用 django cache_page (InvalidCacheBackendError) 时崩溃

Question

我的 Django webapp 项目遇到问题。 我正在使用 Django、Postgres、Redis 和 Celery 运行容器化环境。 主要是，我想使用 Redis 进行缓存和 Celery 设置实时更新。 到目前为止，我已经能够连接到 redis 和 celery 并将 celery 任务结果存储在 Z86A630B907D5EBF183E 中。 当我尝试使用 Redis 缓存 django 中的页面时，事情变得一团糟。 出于某种原因，使用 django 的缓存系统（ cache_page装饰器）会破坏我的 celery 容器。

错误

我的 Celery 容器遇到此错误：

django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django_redis.cache.RedisCache': No module named 'django_redis'

这是完整的回溯：

Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 50, in _create_cache

backend_cls = import_string(backend)

File "/usr/local/lib/python3.9/site-packages/django/utils/module_loading.py", line 17, in import_string

module = import_module(module_path)

File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'django_redis'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/usr/local/bin/celery", line 8, in <module>

sys.exit(main())

File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main

sys.exit(_main())

File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main

return celery(auto_envvar_prefix="CELERY")

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__

return self.main(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main

rv = self.invoke(ctx)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke

return callback(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func

return f(get_current_context(), *args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 132, in caller

return f(ctx, *args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/celery/bin/worker.py", line 320, in worker

worker = app.Worker(

File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 94, in __init__

self.app.loader.init_worker()

File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 111, in init_worker

self.import_default_modules()

File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 105, in import_default_modules

raise response

File "/usr/local/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send

response = receiver(signal=self, sender=sender, **named)

File "/usr/local/lib/python3.9/site-packages/celery/fixups/django.py", line 82, in on_import_modules

self.worker_fixup.validate_models()

File "/usr/local/lib/python3.9/site-packages/celery/fixups/django.py", line 121, in validate_models

run_checks()

File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 70, in run_checks

new_errors = check(app_configs=app_configs, databases=databases)

File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config

return check_resolver(resolver)

File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver

return check_method()

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 408, in check

for pattern in self.url_patterns:

File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__

res = instance.__dict__[self.name] = self.func(instance)

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 589, in url_patterns

patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__

res = instance.__dict__[self.name] = self.func(instance)

File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 582, in urlconf_module

return import_module(self.urlconf_name)

File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1030, in _gcd_import

File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 680, in _load_unlocked

File "<frozen importlib._bootstrap_external>", line 790, in exec_module

File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

File "/code/footballdata/urls.py", line 23, in <module>

from api import views

File "/code/api/views.py", line 21, in <module>

from .mixins import CacheMixin

File "/code/api/mixins.py", line 10, in <module>

class CacheMixin(object):

File "/code/api/mixins.py", line 16, in CacheMixin

def list(self, request):

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 63, in _dec

return _multi_decorate(decorator, obj)

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 47, in _multi_decorate

_update_method_wrapper(_wrapper, dec)

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 17, in _update_method_wrapper

def dummy(*args, **kwargs):

File "/usr/local/lib/python3.9/site-packages/django/utils/decorators.py", line 117, in _decorator

middleware = middleware_class(view_func, *m_args, **m_kwargs)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 174, in __init__

super().__init__(get_response)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 67, in __init__

super().__init__(get_response)

File "/usr/local/lib/python3.9/site-packages/django/middleware/cache.py", line 133, in __init__

self.cache = caches[self.cache_alias]

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 79, in __getitem__

cache = _create_cache(alias)

File "/usr/local/lib/python3.9/site-packages/django/core/cache/__init__.py", line 52, in _create_cache

raise InvalidCacheBackendError(

django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django_redis.cache.RedisCache': No module named 'django_redis'

设置

将 Django 与 django-redis==4.12.1 和 redis==3.5.3 一起使用。 使用 docker-compose 进行 Docker 化。

我的docker-compose.yml ：

version: "3.9"
   
services:
  postgresdb:
    image: postgres
    container_name: postgresdb
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes: 
      - pgdata:/code/pgdata/

  redisdb:
    image: redis
    container_name: redisdb
    volumes: 
      - redisdata:/code/redisdata/

  web:
    build: .
    container_name: django
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    environment: 
      - CELERY_BROKER=redis://redisdb:6379/0
      - CELERY_BACKEND=redis://redisdb:6379/0
    ports:
      - "8000:8000"
    depends_on:
      - postgresdb
      - redisdb

  celery:
    build: .
    container_name: celery
    command: celery -A footballdata worker -l INFO
    volumes: 
      - .:/code
    depends_on: 
      - web
      - redisdb

volumes: 
  pgdata:
  redisdata:

在我的settings.py ：

        CACHES = {
        "default": {
            "BACKEND": "django_redis.cache.RedisCache",
            "LOCATION": "redis://redisdb:6379/",
            "OPTIONS": {
                "CLIENT_CLASS": "django_redis.client.DefaultClient",
            }
        }
    }

CACHE_TTL = 30

# Celery stuff 
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", "redis://redisdb:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", "redis://redisdb:6379/0")
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Warsaw'

# Redis 
REDIS_HOST = 'redisdb'
REDIS_PORT = 6379

是什么导致容器崩溃

使用 django 的cache_page装饰器。 我有一个 REST API 我想连接到缓存。 我使用简单的 mixin 缓存页面：

CACHE_TTL = getattr(settings, 'CACHE_TTL', DEFAULT_TIMEOUT)

class CacheMixin(object):
    """
    Caching for REST API views.
    Caches list and retrieve results of the view for the specified TTL.
    """
    @method_decorator(cache_page(CACHE_TTL))
    def list(self, request):
        return super().list(request)

    @method_decorator(cache_page(CACHE_TTL))
    def retrieve(self, request, pk):
        return super().retrieve(request, pk)

使用这个 mixin 会破坏我的 celery 容器。 当我注释掉这段代码时，容器运行得很好。

据我所知，服务器尝试创建或连接到我在settings.py中指定的缓存后端，但没有找到 django_redis 模块。 这很奇怪，因为我已经安装了它并在我的requirements.txt文件中。

更奇怪的是（至少对我来说）其他容器工作得很好——而且缓存混合也工作，当我连接到 redis 容器并使用这个混合缓存页面时。 它已正确存储在缓存中。 只是 Celery 由于某种原因坏了。

我试过的

1. 四处寻找类似的问题。 我发现的最接近的是这个问题。 但是，此问题与six package 相关。 我的回溯是不同的。 除此之外，我还没有找到任何对我有帮助的东西。

2. 恢复到旧版本的django-redis (3.6.6) 和/或 redis (2.10) - 我看到有些人在过去的 django 和 Z86A1B907D54BF7310394ZBF3167 中存在兼容性问题。 这对我也不起作用。

有没有人有类似的问题？ 我会很感激任何想法。

Answer 1

好的，我解决了。 结果比我想象的要简单（通常是这样）。 我应该只是听了错误消息并知道它来自哪里。

发生的事情是我对我的需求文件进行了一些更改，但我没有使用docker-compose up --build重建容器，而是使用docker build. . 所以我猜 celery 容器没有正确更新所需的包。

所以这个错误是由于我对 docker vs docker-compose 缺乏了解，本质上。