ImportError with Django 1.8.5 and Celery

Question

I'm trying to get Celery to work Django, following the official documentation, and the video located here: https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/

I don't know what I'm doing wrong, but this always results in an ImportError.

The project is called "clubmgmt"

Contents of "clubmgmt/celery.py" :

from __future__ import absolute_import
import os
import django
from celery import Celery
from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'clubmgmt.settings')
django.setup()

app = Celery('clubmgmt')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Contents of "clubmgmt/__init__.py"

from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from clubmgmt.celery import app as celery_app

the task is defined in an app called "activation"

Contents of "activation/tasks.py":

from activation.models import Activation
from django.conf import settings
from django.template.loader import render_to_string
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from clubmgmt.celery import app

__author__ = 'jeroenjacobs'

@app.task
def send_activation_mail(activation_pk):
    activation = Activation.objects.get(pk=activation_pk)
    mail = activation.user.email

    msg = MIMEMultipart('alternative')
    msg['Subject'] = "Please activate your account"
    msg['From'] = settings.SMTP_SENDER_ADDRESS
    msg['To'] = mail
    ...

I always receive the following error:

(clubmgmt) > $ python manage.py runserver                                                                                                                      [±master ●●●]
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/core/management/__init__.py", line 303, in execute
    settings.INSTALLED_APPS
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/jeroenjacobs/.pyenv/versions/3.4.3/Python.framework/Versions/3.4/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2212, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/clubmgmt/__init__.py", line 5, in <module>
    from clubmgmt.celery import app as celery_app
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/clubmgmt/celery.py", line 9, in <module>
    django.setup()
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/jeroenjacobs/.pyenv/versions/clubmgmt/lib/python3.4/site-packages/django/apps/registry.py", line 115, in populate
    app_config.ready()
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/activation/apps.py", line 10, in ready
    import activation.handlers
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/activation/handlers.py", line 6, in <module>
    from activation.utils import create_or_update_activation
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/activation/utils.py", line 2, in <module>
    from activation.tasks import send_activation_mail
  File "/Users/jeroenjacobs/PycharmProjects/clubmgmt/activation/tasks.py", line 7, in <module>
    from clubmgmt.celery import app
ImportError: cannot import name 'app'

Can someone please explain me why "app" cannot be imported?

BTW: This is tested under python 3.4. Going back to Python 2 is not an option, so please don't suggest this as a solution. If Celery is not Python3 compatible, are there any alternatives?

Answer 1

The problem is that clubmgmt.celery.app tries to import clubmgm.celery.app resulting in a circular import.

If you look at your stack trace clubmgmt.__init__ imports clubmgmt.celery.app , which in turns makes django run its setup routine which ends up triggering the import of clubmgmt.activation.apps which at some point through clubmgmt.activation.tasks imports clubmgmt.celery.app again, while it's still being imported, resulting in an ImportError .

Removing that import from __init__.py will definitely solve the problem, but if you still need it there, you might be able to do something with activation/apps.py instead.

One thing I've learned so far is that 90% of the time ImportError occurs due to circular imports, the other 10% is due to misspelling, or something like that.