Celery worker details

Question

I have celery task with 100 input data in queue and need to execute using 5 workers.

• How can I get which worker is executing which input?
• Each worker executed how many inputs and its status?
• If any task is failed how can get failed input data in separately and re-execute with available worker?

Is there any possible ways to customize celery based on worker specific.

We can combine celery worker limitation and flower

I am not using any framework.

Answer 1

How can I get which worker is executing which input?

There are 2 options to use multiple workers:

1. You run each worker separately with separate run commands
2. You run in one command using command line option -c ie concurrency

First method, flower will support it and will show you all the workers, all the tasks (you call inputs), which worker processed which task and other information too.

With second method, flower will show you all the tasks being processed by single worker. In this case you can only differentiate by viewing logs generated by celery worker as in logs it does store which worker THREAD executed which task. So, i think you will be better using first option given your requirements.

Each worker executed how many inputs and its status?

As I mentioned, using first approach, flower will give you this information.

If any task is failed how can get failed input data in separately and re-execute with available worker?

Flower does provide the filters to filter the failed tasks and does provide what status tasks returned when exiting. Also you can set how many times celery should retry a failed task. But even after retries task fails, then you will have to relaunch the task yourself.

Answer 2

For the first and second question:

1) Using Flower API:
You can use celery flower to keep track of it. Flower api can provide you the information like which task is being executed by which worker through simple api calls (/api/task/info/<task_id>) 

2) Querying celery directly:
from celery import Celery
celery = Celery('vwadaptor', broker='redis://workerdb:6379/0',backend='redis://workerdb:6379/0')
celery.control.inspect().active()

3) Using celery events:
   Link : http://docs.celeryproject.org/en/latest/userguide/monitoring.html
   (look Real-time Processing)
   You can create an event  ( task created, task received, etc) and the response will have the worker name(hostname , see the link)

For the third question:
Use the config entry 'CELERY_ACKS_LATE=True' to retry failed tasks.
celery.conf.update(
    CELERY_ACKS_LATE=True,
)

You can also track failed tasks using celery events mentioned above and retry failed tasks manually.