Python Django RabbitMQ芹菜导入任务问题

Question

Welcome... I'm creating a project where I parse xlsx files with xlrd library. Everything works just fine. Then I configured RabbitMQ and Celery. Created some tasks in main folder which works and can be accessed from iPython. The problems starts when I'm in my application (application created back in time in my project) and I try to import tasks from my app in my views.py I tried to import it with all possible paths but everytime it throws me an error. Official documentation posts the right way of importing tasks from other applications, It looks like this: from project.myapp.tasks import mytask But it doesn't work at all. In addition when Im in iPython I can import tasks with command from tango.tasks import add And it works perfectly.

Just bellow I'm uploading my files and error printed out by console.

views.py

# these are the instances that I was trying to import that seemed to be the most reasonable, but non of it worked
# import tasks
# from new_tango_project.tango.tasks import add
# from new_tango_project.tango import tasks
# from new_tango_project.new_tango_project.tango.tasks import add
# from new_tango_project.new_tango_project.tango import tasks
# from tango import tasks

#function to parse files
def parse_file(request, file_id):
    xlrd_file = get_object_or_404(xlrdFile, pk = file_id)
    if xlrd_file.status == False
        #this is some basic task that I want to enter to
        tasks.add.delay(321,123)

settings.py

#I've just posted things directly connected to celery
import djcelery
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'tango',
    'djcelery',
    'celery',
)

BROKER_URL = "amqp://sebrabbit:seb@localhost:5672/myvhost"
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "myvhost"
BROKER_USER = "sebrabbit"
BROKER_PASSWORD = "seb"
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT=['json']
CELERY_TIMEZONE = 'Europe/Warsaw'
CELERY_ENABLE_UTC = False

celery.py (in my main folder new_tango_project )

from __future__ import absolute_import
import os
from celery import Celery
import djcelery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'new_tango_project.settings')

app = Celery('new_tango_project')

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

# CELERY_IMPORTS = ['tango.tasks']
# Optional configuration, see the application user guide.
app.conf.update(
    CELERY_TASK_RESULT_EXPIRES=3600,
    CELERY_RESULT_BACKEND='djcelery.backends.cache:CacheBackend',
)

if __name__ == '__main__':
    app.start()

tasks.py (in my main project folder new_tango_project )

from __future__ import absolute_import
from celery import Celery
from celery.task import task

app = Celery('new_tango_project',
             broker='amqp://sebrabbit:seb@localhost:5672/myvhost',
             backend='amqp://',
             include=['tasks'])

@task
def add(x, y):
    return x + y


@task
def mul(x, y):
    return x * y


@task
def xsum(numbers):
    return sum(numbers)

@task
def parse(file_id, xlrd_file):

    return "HAHAHAHHHAHHA"

tasks.py in my application folder

from __future__ import absolute_import
from celery import Celery
from celery.task import task    
#
app = Celery('tango')

@task
def add(x, y):
    return x + y

@task
def asdasdasd(x, y):
    return x + y

celery console when starting

 -------------- celery@debian v3.1.17 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.8
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         new_tango_project:0x1b746d0
- ** ---------- .> transport:   amqp://sebrabbit:**@localhost:5672/myvhost
- ** ---------- .> results:     amqp://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

Finally my console log...

[2015-02-20 11:19:45,678: ERROR/MainProcess] Received unregistered task of type 'new_tango_project.tasks.add'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'utc': True, 'chord': None, 'args': (123123123, 123213213), 'retries': 0, 'expires': None, 'task': 'new_tango_project.tasks.add', 'callbacks': None, 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, 'id': 'd9a8e560-1cd0-491d-a132-10345a04f391'} (233b)
Traceback (most recent call last):
  File "/home/seb/PycharmProjects/tango/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 455, in on_task_received
    strategies[name](message, body,
KeyError: 'new_tango_project.tasks.add'

This is the log from one of many tries importing the tasks.

Where I`m making mistake ?

Best wishes

Answer 1

Hint 1: In all your tasks.py you declare your Celery app as app = Celery(...) but you don't specify which app the task should be attached to in your task decorators.

Try to change your @task into @app.task and see if it works.

Hint 2: Why do you need to create a new Celery app in every tasks.py ? Why don't you just import one main Celery app with from new_tango_project.celery import app and then declare your tasks with @app.task ?

Hint 3: Once you have your tasks defined (possibly both in celery.py and tasks.py in the applications), just do

from new_tango_project.celery import add
from my_app.tasks import add_bis

def my_view(request):
    ...
    add.delay(*your_params)   # using the task from your celery.py
    add_bis.delay(*your_params) # your task from the application

Answer 2

I wonder how you start up your celery worker. I encounter this once because I didn't start worker right: You should add -A option when execute "celery worker -l info" so that celery will connect to the broker you configured in your Celery Obj. Otherwise celery will try to connect the default broker.