Scheduling a task with Django, celery and Redis

Question

I added a task in Celery for data exportation. I want that when a user calls on the Export all button, the task of exportation starts and there is a django message shown on the frontend while the exportation continue in the back ground. But from what I'm able to do, I have this in views.py :

def CasesPaluExport(request):
    export_cases_palu.delay(request=request)
    messages.success(request, 'CasesPaluExport')
    return JsonResponse({"Ok": "ok"}, safe=False)

and this in tasks.py :

def export_cases_palu(request):
    try:
        plaintext = get_template('stock/download_case_palu.txt')
        htmly = get_template('stock/download_case_palu.html')
        d = Context({'username': request.user.username})

        subject, from_email, to = 'Download all cases palu', settings.DEFAULT_FROM_EMAIL, request.user.email
        text_content = plaintext.render(d)
        html_content = htmly.render(d)
        msg = EmailMultiAlternatives(subject, text_content, from_email, [to])
        msg.attach_alternative(html_content, "text/html")
        msg.send()
    except:
        logging.warning("Tried to send download email to user {0}".format(request.user))

The problem is that whenever I call that CasesPaluExport function, I get an error at the export_cases_palu.delay(request=request) line:

Internal Server Error: /user/casepaluexport/
Traceback (most recent call last):
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/user/project/user/views.py", line 170, in CasesPaluExport
    export_cases_palu.delay(request=request)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/celery/app/task.py", line 413, in delay
    return self.apply_async(args, kwargs)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/celery/app/task.py", line 536, in apply_async
    **options
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/celery/app/base.py", line 737, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/celery/app/amqp.py", line 554, in send_task_message
    **properties
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/utils/json.py", line 72, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/home/user/.virtualenvs/project/local/lib/python2.7/site-packages/kombu/utils/json.py", line 62, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
EncodeError: <WSGIRequest: POST '/user/casepaluexport/'> is not JSON serializable
[20/Nov/2017 15:47:56] "POST /user/casepaluexport/ HTTP/1.1" 500 186320

Which may indicate that the server is still waiting for a http response. How can I solve that ?

Answer 1

The error is telling you that it can't serialize the entire request to send it to the task.

But the only thing you use in the task itself from the request is the user object. You should just send the user ID, and get the object itself inside the task.

 export_cases_palu.delay(user_id=request.user.id)

...

def export_cases_palu(user_id):
    user = User.objects.get(id=user_id)
    d = Context({'username': user.username})

(Also, you shouldn't wrap the whole task in a try/except like that.)

Answer 2

1st thing: You have to restart celery every time when you are adding or removing any method to celery
2nd thing: celery internally stores(stacks) method call in a database whith their parameters in JSON object format so you can pass only JSON serializable objects there and request is not JSON serializable