Celery truncates log messages

Question

My Celery log consistently truncates (not very) long error messages, like this:

[2012-04-08 04:53:10,084: INFO/MainProcess] Task mainapp.tasks.async_submitter[2df2fe93-156b-4944-9ecf-c55ba53e8aaa] succeeded in 0.190640926361s: 'An error occurred during the submission of...

Needless to say, this removes half the purpose of logging. How can I stop this from happening?

I'm running celery with django-celery (through django-supervisor with supervisor) on linux.

Answer 1

Unfortunately, Celery will truncate messages by default yes.

From version 3.1.7 to 3.1.9 it seems to be possible to tweak this limit by patching a module's global:

import celery.worker.job

celery.worker.job.RESULT_MAXLEN = 1048576  # 1 Mib

Answer 2

Task,return by design, should be used for further processing in your code and this is why probably the developer took the freedom to truncate the log, also because returns can be quite big results of elaboration and could make unreadable a log info output. Off course I could be wrong but since I work with Celery I always found this logic correct and never disturbed me. In your case I think it make sense to Log the message just before the return with logger.info (that will not be truncated) and if the message you are returning is actually not needed for further processing but only for logging purpose just don't return any value.