外部 API RabbitMQ 和 Celery 速率限制

Question

I'm using an external REST API which limits my API request at 1 CPS.

This is the following architecture:

Versions:

• Flask
• RabbitMQ 3.6.4
• AMPQ 1.4.9
• kombu 3.0.35
• Celery 3.1.23
• Python 2.7

API client send Web request to internal API, API process the request and control at which rate is sent to RabbitMQ. These tasks could take from 5 seconds to 120 seconds, and there are situations in which tasks may Queue up and they get sent to external API at a higher rate than the one defined, resulting in numerous failed requests. (Resulting in around 5% of failed requests)

Possible solutions:

• Increase External API limit
• Add more workers
• Keep track of failed tasks and retry them later

Although those solutions may work, is not solving exactly the implementation of my rate limiter and controlling the real rate in which my workers can process the API requests. As later I really need to control the external rate.

I believe if I can control RabbitMQ rate limit in which messages can be sent to workers, this could be a better option. I found the rabbitmq prefetch option but not sure if anyone can recommend other options to control the rate in which messages are sent to consumers?

Answer 1

You will need to create your own rate limiter as Celery's rate limit only works per-worker and "does not work as you expect it to".

I have personally found it completely breaks when trying to add new tasks from another task.

I think the requirement spectrum for rate limiting is too wide and depends on an application itself, so Celery's implementation is intentionally too simple.

Here is an example I've created using Celery + Django + Redis . Basically it adds a convenience method to your App.Task class which will keep track of your task execution rate in Redis . If it is too high, the task will Retry at a later time.

This example uses sending a SMTP message as an example, but can easily be replaced with API calls.

The algorithm is inspired by Figma https://www.figma.com/blog/an-alternative-approach-to-rate-limiting/

https://gist.github.com/Vigrond/2bbea9be6413415e5479998e79a1b11a

# Rate limiting with Celery + Django + Redis
# Multiple Fixed Windows Algorithm inspired by Figma https://www.figma.com/blog/an-alternative-approach-to-rate-limiting/
#   and Celery's sometimes ambiguous, vague, and one-paragraph documentation
#
# Celery's Task is subclassed and the is_rate_okay function is added


# celery.py or however your App is implemented in Django
import os
import math
import time

from celery import Celery, Task
from django_redis import get_redis_connection
from django.conf import settings
from django.utils import timezone


app = Celery('your_app')

# Get Redis connection from our Django 'default' cache setting
redis_conn = get_redis_connection("default")

# We subclass the Celery Task
class YourAppTask(Task):
  def is_rate_okay(self, times=30, per=60):
    """
      Checks to see if this task is hitting our defined rate limit too much.
      This example sets a rate limit of 30/minute.

      times (int): The "30" in "30 times per 60 seconds".
      per (int):  The "60" in "30 times per 60 seconds".

      The Redis structure we create is a Hash of timestamp keys with counter values
      {
        '1560649027.515933': '2',  // unlikely to have more than 1
        '1560649352.462433': '1',
      }

      The Redis key is expired after the amount of 'per' has elapsed.
      The algorithm totals the counters and checks against 'limit'.

      This algorithm currently does not implement the "leniency" described 
      at the bottom of the figma article referenced at the top of this code.
      This is left up to you and depends on application.

      Returns True if under the limit, otherwise False.
    """

    # Get a timestamp accurate to the microsecond
    timestamp = timezone.now().timestamp()

    # Set our Redis key to our task name
    key = f"rate:{self.name}"

    # Create a pipeline to execute redis code atomically
    pipe = redis_conn.pipeline()

    # Increment our current task hit in the Redis hash
    pipe.hincrby(key, timestamp)

    # Grab the current expiration of our task key
    pipe.ttl(key)

    # Grab all of our task hits in our current frame (of 60 seconds)
    pipe.hvals(key)

    # This returns a list of our command results.  [current task hits, expiration, list of all task hits,]
    result = pipe.execute()

    # If our expiration is not set, set it.  This is not part of the atomicity of the pipeline above.
    if result[1] < 0:
        redis_conn.expire(key, per)

    # We must convert byte to int before adding up the counters and comparing to our limit
    if sum([int(count) for count in result[2]]) <= times:
        return True
    else:
        return False


app.Task = YourAppTask
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

...

# SMTP Example
import random
from YourApp.celery import app
from django.core.mail import EmailMessage

# We set infinite max_retries so backlogged email tasks do not disappear
@app.task(name='smtp.send-email', max_retries=None, bind=True)
def send_email(self, to_address):

    if not self.is_rate_okay():
        # We implement a random countdown between 30 and 60 seconds 
        #   so tasks don't come flooding back at the same time
        raise self.retry(countdown=random.randint(30, 60))

    message = EmailMessage(
        'Hello',
        'Body goes here',
        'from@yourdomain.com',
        [to_address],
    )
    message.send()