Run a chord callback even if the main tasks fail

Question

Is it possible to run a chord callback even if the main tasks failed?

I've created a chord which I added a bunch of tasks and registered a callback to it. My problem is that if one of the tasks fail, the callback is not triggered. But I would like the callback to be triggered either way.

I've tried to register the callback with si() ( immutability )

callback = tasks.run_delete_rule.si([timestamp])
header = [tasks.run_update_rule.s(i, timestamp) for i in item_ids]
result = chord(header)(callback)

I also tried to add the parameter ignore_result=True to both tasks decorator, but no success.

Answer 1

From the github issue #1881 if the callback has the link_error option set, which takes a list of tasks names, then when a task of the chord fails the link_error tasks are going to be executed.

@task(name='super_task.good')
def good():
    return True

@task(name='super_task.raise_exception')
def raise_exception():
    raise ValueError('error')

@task(name='super_task.callback')
def callback(*args, **kwargs):
    logger.info('callback')
    logger.info(args)
    logger.info(kwargs)
    return 'finished'

@task(name='super_task.error_callback')
def error_callback(*args, **kwargs):
    logger.info('error_callback')
    logger.info(args)
    logger.info(kwargs)
    return 'error'

>>> c = chord(
        [raise_exception.s(), good.s(), raise_exception.s()], 
        callback.s().set(link_error=['super_task.error_callback'])
    )
>>> result = c()

This will execute the chord and in your celery log, you'll see the raise_exception task fail, and the execution of error_callback which will receive in it's args the task_id of callback .

At this point the value of result will contain the AsyncResult instance of callback , and because in a chord the errors propagate to the callback doing result.get() will raise the exception of the tasks and result.traceback gives you the traceback.

If you want to have a single callback, just pass the name of the chord callback to link_error

callback.s().set(link_error='super_task.callback')

NOTE

Another options it's to set CELERY_CHORD_PROPAGATES = False which will revert to the pre celery 3.1 behavior and always execute the callback.

But this is not a recommended approach because as you can find in the github issue #1349

Celery 3.1 defines how chord errors are handled, the previous behavior was never documented and more of an accident since it was never the intention to work that way.

We couldn't change the behavior in a bugfix release so a setting had to be used instead, but it was never the intention that someone would deliberately disable the new behavior.

The new behavior is there to protect against this sort of issue happening, and the backward compatible setting may be removed. I suggest you find some other way to handle errors here (and I wouldn't mind a proposal if you can invent a nice api for it)

Answer 2

You just have to change manner how link_error is called. Instead of string reference, pass signature with arguments you want.

In example above, you can pass argument as following

c = chord(
    [raise_exception.s(), good.s(), raise_exception.s()], 
    callback.s().set(link_error=[error_callback.s(<arguments_here>)])
)

Just keep on mind, first argument will be task_id and further arguments will be those defined in signature.