How to deploy a Django application with the Celery task on Heroku Free plan

Question

I need to deploy my Django application which uses Celery on Heroku. The website is already deployed and works fine, but the Celery worker cannot start.

Here is the output of the heroku logs -t -p celery command:

2020-05-20T16:37:23.529208+00:00 heroku[celery.1]: State changed from starting to up
2020-05-20T16:37:26.576179+00:00 heroku[celery.1]: State changed from up to crashed
2020-05-20T16:37:26.580191+00:00 heroku[celery.1]: State changed from crashed to starting
2020-05-20T16:37:26.438750+00:00 app[celery.1]: Error:
2020-05-20T16:37:26.438775+00:00 app[celery.1]: Unable to load celery application.
2020-05-20T16:37:26.438776+00:00 app[celery.1]: Module 'forum' has no attribute 'celery'

My Procfile :

web: gunicorn --pythonpath forum forum.wsgi --log-file -
celery: celery worker -A forum -l info -c 4

forum/celery.py file:

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'forum.settings')

app = Celery('forum')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

forum/__init__.py file:

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

Inside the forum/settings.py file I have the following parameters related to Celery:

# REDIS related settings
REDIS_HOST = 'localhost'
REDIS_PORT = '6379'
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}

# for Heroku
CELERY_BROKER_URL = os.environ.get('REDIS_URL', 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0')
CELERY_RESULT_BACKEND = os.environ.get('REDIS_URL', 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0')

My Django project ( forum ) has several apps, and the registration app is where my Celery task is located. So, the project has registration/tasks.py file:

import logging

from forum.celery import app


@app.task
def send_verification_email(user_id):
    pass

In registration/signals.py I have the function for sending emails:

from django.db.models.signals import post_save
from django.contrib.auth.models import User
from django.dispatch import receiver

from .tasks import send_verification_email


# send verification email when new user is registered
@receiver(post_save, sender=User)
def user_post_save(sender, instance, created, *args, **kwargs):
    if created:
        # Send verification email
        send_verification_email.delay(instance.pk)

I'm going to use free Heroku dynos (since this is just my pet project for learning). But I suspect that the reason is not in the Heroku pricing plan but in the wrong configuration of the Procfile .

Heroku Redis addon is up and running.

However, I tried to change the Procfile in many ways, but the result is the same - Celery worker cannot start, it crashes. On my local machine Celery works perfectly. I run Celery by the following command (assuming that I started Redis server earlier by the redis-server command):

celery worker -A forum --loglevel=debug --concurrency=4

Can somebody help me, please, with this issue?

Answer 1

The problem was with the wrong structure of the entire Django application. I had something similar to this:

pet
|-- forum
|   |-- forum
|   |   |-- __init__.py
|   |   |-- celery.py
|   |   |-- wsgi.py
|   |   |-- asgi.py
|   |   |-- settings.py
|   |   |-- urls.py
|   |-- app1
|   |-- app2
|   |-- appN
|   |-- manage.py
|   |-- Procfile
|   `-- requirements.txt

The pet/forum/ folder was redundant. The forum folder that contains files as settings.py or celery.py should be directly placed under the pet folder, on the same level as folders for all other apps.

Also, after these changes I needed to modify the Procfile :

web: gunicorn forum.wsgi --log-file -
celery: celery worker -A forum -l info -c 4