Can't get Celery to work on digital ocean droplet with prod settings

Question

It is working fine on local server, but when I try to start a worker after ssh I get an error.

/var/www/bin/celery -A stock worker -l info

I know DJANGO_SETTINGS_MODULE is set correctly as I have a print statement showing it is set, (and the rest of the server is live, using production settings). I've also tried using this command, which gives the same error.

DJANGO_SETTINGS_MODULE="stock.settings.pro" /var/www/bin/celery -A stock worker -l info

I have a celery directory that is in my main directory (beside my settings directory). It contains an init .py file and a conf.py (that sets the results backend). Here is the init file:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'stock.settings.pro')
DJANGO_SETTINGS_MODULE = os.environ.get('DJANGO_SETTINGS_MODULE')
print("celery - ", DJANGO_SETTINGS_MODULE)
BASE_REDIS_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379')

app = Celery('stock')
app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

app.conf.broker_url = BASE_REDIS_URL

app.conf.beat_scheduler = 'django_celery_beat.schedulers:DatabaseScheduler'

Here is the traceback error:

Traceback (most recent call last):
File "/var/www/lib/python3.8/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'data'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/www/bin/celery", line 8, in <module>
sys.exit(main())
File "/var/www/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
_main()
File "/var/www/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/var/www/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/var/www/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/var/www/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
return self.execute(command, argv)
File "/var/www/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
return cls(
File "/var/www/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
*self.parse_options(prog_name, argv, command))
File "/var/www/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
self.parser = self.create_parser(prog_name, command)
File "/var/www/lib/python3.8/site-packages/celery/bin/base.py", line 444, in create_parser
self.add_arguments(parser)
File "/var/www/lib/python3.8/site-packages/celery/bin/worker.py", line 278, in add_arguments
default=conf.worker_state_db,
File "/var/www/lib/python3.8/site-packages/celery/utils/collections.py", line 134, in __getattr__
return self[k]
File "/var/www/lib/python3.8/site-packages/celery/utils/collections.py", line 444, in __getitem__
return getitem(k)
File "/var/www/lib/python3.8/site-packages/celery/utils/collections.py", line 287, in __getitem__
return mapping[_key]
File "/usr/lib/python3.8/collections/__init__.py", line 1006, in __getitem__
if key in self.data:
File "/var/www/lib/python3.8/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/var/www/lib/python3.8/site-packages/celery/app/base.py", line 141, in data
return self.callback()
File "/var/www/lib/python3.8/site-packages/celery/app/base.py", line 920, in _finalize_pending_conf
conf = self._conf = self._load_config()
File "/var/www/lib/python3.8/site-packages/celery/app/base.py", line 930, in _load_config
self.loader.config_from_object(self._config_source)
File "/var/www/lib/python3.8/site-packages/celery/loaders/base.py", line 131, in config_from_object
self._conf = force_mapping(obj)
File "/var/www/lib/python3.8/site-packages/celery/utils/collections.py", line 54, in force_mapping
if isinstance(m, (LazyObject, LazySettings)):
File "/var/www/lib/python3.8/site-packages/django/utils/functional.py", line 224, in inner
self._setup()
File "/var/www/lib/python3.8/site-packages/django/conf/__init__.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/var/www/lib/python3.8/site-packages/django/conf/__init__.py", line 161, in __init__
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

Thanks a lot for any help, I've tried solving this for a couple days now and haven't really gotten anywhere!

Answer 1

I randomly came across your questions when looking for something else. The problem as per your log is obviously: " The SECRET_KEY setting must not be empty. "

At the time when celery runs your secret key is not yet 'present' in your environment it seems. Do you have the same in development? For testing try to set a random Secret key on top of your settings.py file and/or top of your celery init file as you have above. If that issue goes away then you know what the problem is. You can fix it from there then.

Usually the SECRET_KEY should be part of your environment and populated before celery is running. In that case your run order might be wrong or the env vars are not set correctly (or after celery is initialized).

I hope that makes sense.

In my own development (NOT production) environment I have all my vars including the SECRET_KEY in an.env file and would come across the same issue. Basically the celery init.py file will load before SECRET_KEY is initialized. To circumvent that problem I simply put a flag on top of my settings.py file along the lines of:

if development:
   import dotenv
   dotenv.load_dotenv()

Where I set 'development' to True or False depending on what environment it is. Again: if this error happens in production something is wrong with your initialization order most likely.