芹菜非阻塞客户端

Question

import proj.tasks
import time
import sys
import socket
import logging
import datetime

lat_to, ts = proj.tasks.timeme(time.time())          <---- blocking call
lat_from = time.time() - ts
print lat_to, lat_from

Celery task blocks so I cant take advantage of many workers. Is it possible to make that a non blocking call?

NOTE: Ive looked at tornado-celery as an option for non blocking celery client but I am not sure if i like that approach as i need to launch tornado celery web server.

Answer 1

When calling a celery task the method executes synchronously. THe power of a task queue is putting a task on the queue and letting the workers asynchronously do their work.

You can do this using the task. delay method.

I'm not quiet sure what delay does internally but it returns very quickly, and the work of your method is not actually being done when you call it, your task is just being put on the work queue.

Answer 2

tornado-celery works fine on my side, but it by default waits for task's result before callback,

class GenAsyncHandler(web.RequestHandler):
    @asynchronous
    @gen.coroutine
    def get(self):
        response = yield gen.Task(tasks.sleep.apply_async, args=[3])
        self.write(str(response.result))
        self.finish()

if you want to have task callback options as below, you can try my fork

1. After task sent
2. After task sent and ack-ed
3. To fit original celery behavior that task.apply_async() to get the AsyncResult first, then AsyncResult.get() to get actual task result in tornado asynchronous fashion