Celery, task called from other task is not working

Question

I'm trying to make application with cellery. It should work on few workers and different workers are consuming from different queues. I've got something like this:

@celery.task
def task1():
    do_something()
    task2.delay()

@celery.task
def task2()
    do_something()

so task1 which is running on worker1 should call task2 which should be send to queue from which consuming worker2. Problem is that it is not working. I receive id of AsyncResult but state of this task is all time PENDING. When I call task2 manually from python console it works fine. Maybe I'm doing something wrong and it is not possiblem to run one task from other one? And one more thing. Worker1 is doing task1 and send task2 to queue from which he not consuming - from this queue is consuming only worker2

Answer 1

Here's a simple example that I think accomplishes what you want.

from celery import Celery
import random
import string

celery = Celery('two_q',backend='amqp',broker='amqp://guest@localhost//')

@celery.task
def generate_rand_string(n):
    # n = number of characters
    rand_str = "".join([random.choice(string.lowercase) for i in range(n)])
    #calls the second task and adds it to second queue
    reverse.apply_async((rand_str,),queue="q2")
    print rand_str
    return rand_str

@celery.task
def reverse(s):
    print s[::-1]
    return s[::-1]

generate_rand_string.apply_async((10,), queue="q1")

When called with the -Q argument which specifies the list of the queues

 celery worker --app=two_q -l info -Q q1,q2

it produces the following output:

pawel@iqmxma82x7:~/py/celery$ celery worker --app=two_q -l info -Q q1,q2

 -------------- celery@iqmxma82x7 v3.0.23 (Chiastic Slide)
---- **** ----- 
--- * ***  * -- Linux-3.2.0-54-generic-pae-i686-with-Ubuntu-12.04-precise
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> broker:      amqp://guest@localhost:5672//
- ** ---------- .> app:         cel_group:0x9bfef8c
- ** ---------- .> concurrency: 4 (processes)
- *** --- * --- .> events:      OFF (enable -E to monitor this worker)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> q1:          exchange:q1(direct) binding:q1
                .> q2:          exchange:q2(direct) binding:q2

[Tasks]
  . two_q.generate_rand_string
  . two_q.reverse

[2013-09-15 19:10:35,708: WARNING/MainProcess] celery@iqmxma82x7 ready.
[2013-09-15 19:10:35,716: INFO/MainProcess] consumer: Connected to amqp://guest@127.0.0.1:5672//.
[2013-09-15 19:10:40,731: INFO/MainProcess] Got task from broker: two_q.generate_rand_string[fa2ad56e-c66d-44a9-b908-2d95b2c9e5f3]
[2013-09-15 19:10:40,767: WARNING/PoolWorker-1] jjikjkepkc
[2013-09-15 19:10:40,768: INFO/MainProcess] Got task from broker: two_q.reverse[f52a8247-4674-4183-a826-d73cef1b64d4]
[2013-09-15 19:10:40,770: INFO/MainProcess] Task two_q.generate_rand_string[fa2ad56e-c66d-44a9-b908-2d95b2c9e5f3] succeeded in 0.0217289924622s: 'jjikjkepkc'
[2013-09-15 19:10:40,782: WARNING/PoolWorker-3] ckpekjkijj
[2013-09-15 19:10:40,801: INFO/MainProcess] Task two_q.reverse[f52a8247-4674-4183-a826-d73cef1b64d4] succeeded in 0.0195469856262s: 'ckpekjkijj'

You get two queues(q1,q2) and two workers.

As a comparison if you either call it without -Q argument or with only one queue:

celery worker --app=two_q -l info 

The "reverse" task will not be called, because q2 to which it is added will not be known to celery.

Hope it helps.