Log time on queue of celery tasks

Question

I can use signals to log task execution time, but I would like to log also the time on queue. Is this possible with signals? Which signals should I use?

Answer 1

Task events can be used to monitor and trigger action based on the events of a task. Task-sent, task-received, task-started, task-succeeded, task-failed, task-rejected, task-revoked, task-retried are the task events supported in celery. For more details, please refer this link . To log the time a task is waiting in the queue, get the task created (or added to job queue) time and task started time by using the respective task event handlers. The difference of them will give the waiting time of the job in the queue. Below is a sample python code on how to implement it.

    from celery import Celery
    redis = Redis(host='workerdb', port=6379, db=0)
    taskId_startTime = {}
    taskId_createTime = {}

    def my_monitor():
        app = Celery('vwadaptor', broker='redis://workerdb:6379/0',backend='redis://workerdb:6379/0')
        state = app.events.State()

        def announce_task_received(event):
            state.event(event)
            task = state.tasks.get(event['uuid'])
            taskId_createTime[task.uuid] = task.timestamp 

        def announce_task_started(event):
            state.event(event)
            task = state.tasks.get(event['uuid'])
            taskId_startTime[task.uuid] = task.timestamp 

        def announce_task_succeeded(event):
            state.event(event)
            task = state.tasks.get(event['uuid'])
            print "wait time in queue", taskId_startTime[task.uuid] - taskId_createTime[task.uuid] 

        with app.connection() as connection:
            recv = app.events.Receiver(connection, handlers={
                    'task-received': announce_task_received, 
                    'task-started': announce_task_started, 
                    'task-succeeded': announce_task_succeeded, 
            })
            recv.capture(limit=None, timeout=None, wakeup=True)


    my_monitor()