Pass a list to a celery worker with Django

Question

I used Django 1.9 / Python 2.7 / Celery 3.1.23 / Redis 2.10.5

Celery works fine for many simple tasks, but when I am trying to pass lists to my Celery worker, it does not work. The general objective is to lighten the worker process by succesively transmitting chunks of ids to the worker, and not 30,000 all in one time.

I understand I have to pass my list as Json format.

Settings:

CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_SERIALIZER = ['json']
CELERY_TASK_SERIALIZER = ['json']

tasks.py:

@periodic_task(run_every=crontab(minute=29, hour=12))
    def update(request=None):
        iddict = [23, 49, 81, 23]
        forceevaluation = list(sliceids)        
        sliceids2 = json.dumps(forceevaluation)    
        updatewikipediadescription2.apply_async(args=sliceids2, eta=now() + timedelta(seconds=a))

views.py

@shared_task
def updatetask(slice):
    for placeid in slice:
        print("ok")

Celery always shows the following error:

[2017-03-10 12:29:04,031: ERROR/MainProcess] Task googleautocomplete.tasks.updatetask[94cdded2-0b2d-4304-aa14-6d97257c947c] raised unexpected: ValueError('task args must be a list or tuple',) 

Any idea why I get this error ?

Answer 1

You're not passing a list, you're passing the ouput of JSON dumping your list.

sliceids2 = json.dumps(forceevaluation) creates a string:

>> type(sliceids2) <type 'str'> so when you run:

updatewikipediadescription2.apply_async(args=sliceids2, eta=now() + timedelta(seconds=a))

you're just passing a string.

try:

updatewikipediadescription2.apply_async(args=[sliceids2,], eta=now() + timedelta(seconds=a))