在完成所有任务之前执行芹菜和弦

Question

I have a bunch of tasks that can be executed simultaneously, but once everything is ready I want to execute a final one. I am using the following code:

chunk_tasks = []
for index, chunk in enumerate(chunks):
    chunk_tasks.append(import_chunk.s(meta.pk))

g = group(chunk_tasks)
chord(g)(import_completed.s(meta.pk, max_lines=max_lines))

However it looks like import_completed is executed before all tasks are completed. Also the import_chunk task looks like:

@task(bind=True, ignore_result=IGNORE_RESULTS)
def import_chunk(self, meta_pk):
    try:
        # do some stuff
    except Exception, e:
        if self.max_retries == self.request.retries:
            logger.exception('Unexpected error in import_chunk')
        raise self.retry(countdown=60, max_retries=3)

So the question is what am I doing wrong?

Answer 1

Chord is a task that only executes after all of the tasks in a group have finished executing. So, it needs state of tasks in its header for synchronization.

But when you set ignore_result to your task , the worker will not store task state and return values for this task.

This will lead to retrying of task or throwing exception or any malfunction according to your workflow.

So, chord(add.s(i, i) for i in range(10))(tsum.s()).get() is perfectly valid and gives result for CASE 1 but it gives some trouble for CASE 2.

CASE 1:

@app.task
def add(x, y):
    return x + y

@app.task
def tsum(numbers):
    return sum(numbers)

CASE 2:

@app.task(ignore_result=True)
def add(x, y):
    return x + y

@app.task(ignore_result=True)
def tsum(numbers):
    return sum(numbers)

So, you have to change ignore_result or change the workflow of your tasks.

From docs:

You should avoid using chords as much as possible. Still, the chord is a powerful primitive to have in your toolbox as synchronization is a required step for many parallel algorithms.