Using Celery (python) & RabbitMQ to save results of tasks to a specific file location?

Question

Hey guys I'm working on a prototype for a project at my school (I'm a research assistant so this isn't a graded project). I'm running celery on a server cluster (with 48 workers/cores) which is already setup and working. The nutshell of my project is that we want to use celery for some number crunching of a rather large amount of files/tasks.

Because of this it is very important that we save results to an actual file, we have gigs upon gigs of data and it WON'T fit in RAM while running the traditional task queue/backend.

Anyways... My prototype (with a trivial add function):

task.py

from celery import Celery

app=Celery()

@app.task
def mult(x,y):
    return x*y

And this works great when I execute: $ celery worker -A task -l info

But if I try and add a new backend:

from celery import Celery

app=Celery()
app.conf.update(CELERY_RESULT_BACKEND = 'file://~/Documents/results')
@app.task
def mult(x,y):
    return x*y

I get a rather large error:

[2017-08-04 13:22:18,133: CRITICAL/MainProcess] Unrecoverable error: 
AttributeError("'NoneType' object has no attribute 'encode'",)
Traceback (most recent call last):
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bartolucci/anaconda3/lib/python3.6/site-        packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/bootsteps.py", line 115, in start
self.on_start()
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/apps/worker.py", line 143, in on_start
self.emit_banner()
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/apps/worker.py", line 158, in emit_banner
' \n', self.startup_info(artlines=not use_image))),
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/apps/worker.py", line 221, in startup_info
results=self.app.backend.as_uri(),
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/app/base.py", line 1183, in backend
return self._get_backend()
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/app/base.py", line 902, in _get_backend
return backend(app=self, url=url)
  File "/home/bartolucci/anaconda3/lib/python3.6/site-packages/celery/backends/filesystem.py", line 45, in __init__
self.path = path.encode(encoding)
AttributeError: 'NoneType' object has no attribute 'encode'

I am only 2 days into this project and have never worked with celery (or a similar library) before (I come from the algorithmic, mathy side of the fence). I currently wrangling with celery's user guide docs, but they're honestly pretty sparse on this detail.

Any help is much appreciated and thank you.

Answer 1

Looking at the celery code for filesystem backed result backend here. https://github.com/celery/celery/blob/master/celery/backends/filesystem.py#L54

Your path needs to start with file:/// (3 slashes) Your settings has it starting with file:// (2 slashes)

You might also want to use the absolute path instead of the ~.