how can i split up a large django-celery tasks.py module into smaller chunks?

Question

I have some very lengthy task functions that I would like to break out from my tasks.py module, but still be able to call like from myapp.tasks import my_task . I tried creating submodules for each task, but then I have to insert some trickery into tasks/__init__.py , which seems pretty hackish (and requires giving the tasks the same names as the modules):

import pkgutil

for module_loader, name, ispkg in pkgutil.iter_modules(__path__):
    __import__('myapp.tasks.%s' % name)
    locals().update({name: getattr(locals()[name], name)})

Answer 1

This answer is about management commands rather than tasks in the sense of Celery background tasks.

---

I think the most straight-forward solution is to first write the tasks as Django management commands, and then call those from tasks.py. This way you can break up your tasks into one task per file, and have the additional benefit of being able to call your tasks arbitrarily from the command line.

So, in tasks.py :

from django.core import management

@task()
def my_task():
    management.call_command('my_task')

Answer 2

I'm not familiar with Django-Celery but quite familiar with Celery, so this is more of a general note which may answer your question...

Celery tasks are identified by name when executed. By default, tasks are given the name of their Python location, ie. '%s.%s' % (my_task.__module__, my_task.__name__) , or 'myapp.tasks.function_name'. That said, you can override the task name by supplying the @task decorator with the 'name' kwarg. So if you wanted to override a task name:

# We're in myapp.tasks but the task will not be named myapp.tasks.my_task as expected
@task(name='some.overridden.name')
def my_task():
    print 'Something'

Answer 3

The following works as well:

• create a package tasks
• spread your tasks in python modules inside the tasks package
• import the modules in the tasks/__init__.py

Example:

django_app
| tasks
   | __init__.py
   | filesystem_tasks.py
| admin.py
| url.py

• tasks/filesystem_tasks.py can contain this:

   from __future__ import absolute_import, unicode_literals from celery import shared_task @shared_task def add(x, y): return x + y 

• tasks/__init__.py can contain this:

   from .filesystem_tasks import * 

• Running celery indicates the tasks as they would have been declared in one tasks.py at the same level as eg. url.py .