Parallel and Sequential Execution of tasks using Celery

Question

I am pretty new to Celery and planning to use it for scheduling few jobs. One of the scenario is as below.

TASK_NM | DEPENDENCY

T1

T2      |        T1

T3      |        T1 

T4

T5      |        T3 and T4

From the above case,

1. I have to execute T1 and T4 in parallel

2. After Successful completion of T1, execute T2 and T3 in parallel

3. After Successful completion of T3 and T4, execute T5.

If any task failed in the middle, do not execute the dependent jobs.

How can we achieve this using Celery?

Thanks in Advance!!!

Answer 1

Yes, but it will require some work. You will hve to use chain , group , and chord to get the behavior that you want:

1. Execute T1, T2 and T3 serially by using chain :

    res = chain(T1.si(), T2.si(), T3.si())

2. Execute (1) in parallel with T4 by using group :

    res2 = group(res.s(), T4.s())

3. Execute T5 after T3 and T4 are complete by using chord :

    res3 = chord(res2.s(), T5.s())

4. Execute it all

   result = res3().get()

The docs do a pretty good job of explaining which to use for serial and which to use for parallel. The tricky part to remember is that the chain , chord and group all return tasks themselves so have to be invoked using () , delay() or apply_async() .