芹菜，一个再见一个队列，并行许多时间队列

Question

I am trying to use celery to manage tasks. The problem i am into now, that i have many minor tasks(emails, cross-server posts, etc) And time consumable tasks, like file uploads.

Is there any way, to specify, that uploads will always will be one by one. Only one task executed in time, while other workers will work on other queues?

Answer 1

An effective way to serialize the execution of tasks is to use a mutex (Mutual Exclusion).

Python's threading module has a Lock object which can be used to this effect :

# ...
module_lock = threading.Lock() # or make this an attribute in an object with sufficiently-large scope
# ...
def do_interesting_task():
     with module_lock.acquire():
         interesting_task()

"Abandon all hope, ye who enter here."

Mutexes and semaphores are powerful tools, but used unwisely they will yield deadlocks and occasionally eat your lunch.

Answer 2

Meanwhile i've implemented such solution, works pretty good. But, i am not pretty sure, tha max_retries=None states that there are will be unlimited number of retries. This solution works on redis, but can work on any other engine that supports atomicaly operation of increment.

@task(max_retries=None,default_retry_delay=3)
def sleepTask():
    if r.incr('sleep_working')>1:
        r.incr('sleep_working',-1)
        sleepTask.retry()
    else:
        try:
            r.expire('sleep_working',3600)
            sleep(30)
        finally:
            r.incr('sleep_working',-1)
        return True

The key here, is that incr is atomically, so it will never happen, that two clients receive counter==1.

Also expire is very important, anything can happen, and we will get our counter >1 forever, so expire makes sure, that no matter what, after specific time counter will be dropped. This value can be adjusted to the needs. Mine is big files uploads, so 3600 sounds OK.

I think that this is good starting point, to make custom Task object, that will do all of this automatically, by recieving redis_key and expire_time values. I will update this post if I'll do such a Task.

As a bonus, this solution is also can be easily adjusted to 2/3/any other number of parallel limit, by changing >1 to >anynumber