Celery + Python：将耗时的任务排队在另一个任务中

Question

I want to query an api (which is time consuming) with lots of items (~100) but not all at once. Instead I want a little delay between the queries.

What I currently have is a task that gets executed asynchronously and iterates over the queries and after each iteration waits some time:

@shared_task
def query_api_multiple(values):
    delay_between_queries = 1

    query_results = []

    for value in values:
        time.sleep(delay_between_queries)

        response = query_api(value)
        if response['result']:
            query_results.append(response)

    return query_results

My question is, when multiple of those requests come in, will the second request gets executed after the first is finished or while the first is still running? And when they are not getting executed at the same time, how can I achieve this?

Answer 1

You should not use time.sleep but rate limit your task instead:

Task.rate_limit

Set the rate limit for this task type (limits the number of tasks that can be run in a given time frame).

The rate limits can be specified in seconds, minutes or hours by appending “/s”, “/m” or “/h” to the value. Tasks will be evenly distributed over the specified time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of 600ms between starting two tasks on the same worker instance.

So if you want to limit it to 1 query per second, try this:

@shared_task(rate_limit='1/s')
def query_api_multiple(values):
    ...

Answer 2

Yes, if you create multiple tasks then they may run at the same time.

You can rate limit on a task type basis with celery if you want to limit the number of tasks that run per period of time. Alternatively, you could implement a rate limiting pattern using something like redis, combined with celery retries, if you need more flexibility than what celery provides OOtB.