Python Celery 任务优先级未按预期工作

Question

Hi I am trying to send tasks with priority using the Python ThreadPoolExecutor to send 8 tasks concurrently. How do I achieve the expectation ?

I started the worker using the following command.

celery -A task worker --loglevel=info -P eventlet --concurrency=5

Observation : As seen in the screenshot of the logs below, the tasks are running in random order, and not following the priority.

Expectation : The tasks run in order of priority.

task.py

from __future__ import absolute_import
from celery import Celery
import sys
import time
from kombu import Queue, Exchange

# celery_url = "amqp://%s:%s@%s//" % ("guest", "guest", "0.0.0.0")
celery_app = Celery("priority_queue")

celeryconfig = {}
celeryconfig['BROKER_URL'] = 'pyamqp://guest@localhost//'
celeryconfig['CELERY_RESULT_BACKEND'] = 'rpc://'
celeryconfig['CELERY_QUEUES'] = (
    Queue('tasks', Exchange('tasks'), routing_key='tasks', queue_arguments={'x-max-priority': 10}),
)
celeryconfig['CELERY_ACKS_LATE'] = True
celeryconfig['CELERYD_PREFETCH_MULTIPLIER'] = 1

celery_app.config_from_object(celeryconfig)


@celery_app.task(queue='tasks', ignore_result=True)
def add(x, y, priority):
    print("Adding {} and {} with priority {}".format(x, y, priority))
    time.sleep(5)
    print("Result {} and {} with priority {}".format(x, y, priority))
    return x + y

client.py

from __future__ import absolute_import

from concurrent.futures.thread import ThreadPoolExecutor
from testapp import add


def add_with_priority(x, y, priority):
    res = add.apply_async(args=[x, y, priority], routing_key='tasks', priority=priority)
    if res.ready():
        print("Result from {} + {} with priority {} is {}".format(x, y, priority, res))


if __name__ == '__main__':
    test_samples = [[4, 5, 1], [2, 9, 2], [10, 25, 3], [4, 5, 4], [4, 5, 5], [4, 5, 6], [4, 5, 7], [4, 5, 8]]

    with ThreadPoolExecutor(max_workers=4) as e:
        for test in test_samples:
            e.submit(add_with_priority, *test)

Worker logs

Answer 1

I looked in my code (that worked back then):

def build_task(task_funct, queue, priority, args):
    return task_funct.signature(args=args, queue=queue, priority=priority, immutable=True, kwargs={'priority': priority}

compared to your code - maybe you can try to change this line:

res = add.apply_async(args=[x, y, priority], routing_key='tasks', priority=priority)

to:

res = add.apply_async(args=[x, y, priority], queue='tasks', routing_key='tasks', priority=priority, kwargs={'priority': priority})

Please let me know if it makes any different

EDIT:

Can you try it like that:

task = build_task(add, args=(x, y, priority), queue='tasks', routing_key='tasks', priority=priority)
(task).apply_async()

(using my build_task above)