wrapping rate limiting API call

Question

I have access an API call that accepts a maximum rate of calls per second. If the rate is exceeded, an exception is thrown.

I would like to wrap this call into an abstraction that does the necessary to keep the call rate under the limit. It would act like a.network router: handling multiple calls and returning the results to the correct caller caring about the call rate. The goal is to make the calling code as unaware as possible about that limitation. Otherwise, every part in the code having this call would have to be wrapped into a try-catch!

For example: Imagine that you can call a method from an external API that can add 2 numbers. This API can be called 5 times per second. Anything higher than this will result in an exception.

To illustrate the problem, the external service that limits the call rate is like the one in the answer to this question: How to build a rate-limiting API with Observables?

Additional info

Since you don't want the worry about that limit every time you call this method from any part of your code, you think about designing a wrapper method that you could call without worrying about the rate limit. On the inside you care about the limit, but on the outside you expose a simple async method.

It's similar to a web server. How does it return the correct pack of results to the correct customer?

Multiple callers will call this method, and they will get the results as they come. This abstraction should act like a proxy.

How could I do it?

I'm sure the firm of the wrapper method should be like

public async Task<Results> MyMethod()

And inside the method it will perform the logic, maybe using Reactive Extensions (Buffer). I don't know.

But how? I mean, multiple calls to this method should return the results to the correct caller. Is this even possible?

Answer 1

There are rate limiting libraries available (see Esendex's TokenBucket Github or Nuget ).

Usage is very simple, this example would limit polling to 1 a second

// Create a token bucket with a capacity of 1 token that refills at a fixed interval of 1 token/sec.
ITokenBucket bucket = TokenBuckets.Construct()
  .WithCapacity(1)
  .WithFixedIntervalRefillStrategy(1, TimeSpan.FromSeconds(1))
  .Build();

// ...

while (true)
{
  // Consume a token from the token bucket.  If a token is not available this method will block until
  // the refill strategy adds one to the bucket.
  bucket.Consume(1);

  Poll();
}

I have also needed to make it async for a project of mine, I simply made an extension method:

public static class TokenBucketExtensions
{
    public static Task ConsumeAsync(this ITokenBucket tokenBucket)
    {
        return Task.Factory.StartNew(tokenBucket.Consume);
    }
}

Using this you wouldn't need to throw/catch exceptions and writing a wrapper becomes fairly trivial

Answer 2

What exactly you should depends on your goals and limitations. My assumptions:

• you want to avoid making requests while the rate limiter is in effect
• you can't predict whether a specific request would be denied or how exactly will it take for another request to be allowed again
• you don't need to make multiple request concurrently, and when multiple requests are waiting, it does not matter in which order are they completed

If these assumptions are valid, you could use AsyncAutoResetEvent from AsyncEx : wait for it to be set before making the request, set it after successfully making a request and set it after a delay when it's rate limited.

The code can look like this:

class RateLimitedWrapper<TException> where TException : Exception
{
    private readonly AsyncAutoResetEvent autoResetEvent = new AsyncAutoResetEvent(set: true);

    public async Task<T> Execute<T>(Func<Task<T>> func) 
    {
        while (true)
        {
            try
            {
                await autoResetEvent.WaitAsync();

                var result = await func();

                autoResetEvent.Set();

                return result;
            }
            catch (TException)
            {
                var ignored = Task.Delay(500).ContinueWith(_ => autoResetEvent.Set());
            }
        }
    }
}

Usage:

public static Task<int> Add(int a, int b)
{
    return rateLimitedWrapper.Execute(() => rateLimitingCalculator.Add(a, b));
}

Answer 3

A variant to implement this is to ensure a minimum time between calls, something like the following:

private readonly Object syncLock = new Object();
private readonly TimeSpan minTimeout = TimeSpan.FromSeconds(5);
private volatile DateTime nextCallDate = DateTime.MinValue;

public async Task<Result> RequestData(...) {
    DateTime possibleCallDate = DateTime.Now;
    lock(syncLock) {
        // When is it possible to make the next call?
        if (nextCallDate > possibleCallDate) {
            possibleCallDate = nextCallDate;
        }
        nextCallDate = possibleCallDate + minTimeout;
    }

    TimeSpan waitingTime = possibleCallDate - DateTime.Now;
    if (waitingTime > TimeSpan.Zero) {
        await Task.Delay(waitingTime);
    }

    return await ... /* the actual call to API */ ...;
}

Answer 4

Older thread but in this context I would also like to mention the Polly library https://github.com/App-vNext/Polly which can be very helpful in this scenario.

Can do the same (and more) as TokenBucket https://github.com/esendex/TokenBucket mentioned in another answer, but this library is a bit older and doesn't support for example .net standard by default.