Delay function execution (API call) to execute n times in time period

Question

I'm trying to write back end functionality that is handling requests to particular API, but this API has some restrictive quotas, especially for requests/sec. I want to create API abstraction layer that is able of delaying function execution if there are too many requests/s, so it works like this:

1. New request arrives (to put it simple - library method is invoked)
2. Check if this request could be executed right now, according to given limit (requests/s)
3. If it can't be executed, delay its execution till next available moment
4. If at this time a new request arrives, delay its execution further or put it on some execution queue

I don't have any constraints in terms of waiting queue length. Requests are function calls with node.js callbacks as the last param for responding with data.

I thought of adding delay to each request, which would be equal to the smallest possible slot between requests (expressed as minimal miliseconds/request), but it can be a bit inefficient (always delaying functions before sending response).

Do you know any library or simple solution that could provide me with such functionality?

Answer 1

Save the last request's timestamp.

Whenever you have a new incoming request, check if a minimum interval elapsed since then, if not, put the function in a queue then schedule a job (unless one was already scheduled):

setTimeout(
  processItemFromQueue,
  (lastTime + minInterval - new Date()).getTime()
)

processItemFromQueue takes a job from the front of the queue ( shift ) then reschedules itself unless the queue is empty.

Answer 2

The definite answer for this problem (and the best one) came from the API documentation itself. We use it for a couple of months and it perfectly solved my problem.

In such cases, instead of writing some complicated queue code, the best way is to leverage JS possibility of handling asynchronous code and either write simple backoff by yourself or use one of many great libraries to use so.

So, if you stumble upon any API limits (eg quota, 5xx etc.), you should use backoff to recursively run the query again, but with increasing delay (more about backoff could be found here: https://en.wikipedia.org/wiki/Exponential_backoff ). And, if finally, after given amount of times you fail again - gracefully return error about unavailability of the API.

Example use below (taken from https://www.npmjs.com/package/backoff ):

var call = backoff.call(get, 'https://someaddress', function(err, res) {
    console.log('Num retries: ' + call.getNumRetries());

    if (err) {
        // Put your error handling code here.
        // Called ONLY IF backoff fails to help
        console.log('Error: ' + err.message); 
    } else {
        // Put your success code here
        console.log('Status: ' + res.statusCode);
    }
});

/*
 * When to retry. Here - 503 error code returned from the API
 */
call.retryIf(function(err) { return err.status == 503; }); 

/*
 * This lib offers two strategies - Exponential and Fibonacci. 
 * I'd suggest using the first one in most of the cases
 */
call.setStrategy(new backoff.ExponentialStrategy());

/*
 * Info how many times backoff should try to post request
 * before failing permanently
 */
call.failAfter(10);

// Triggers backoff to execute given function
call.start();

There are many backoff libraries for NodeJS, leveraging either Promise-style, callback-style or even event-style backoff handling (example above being second of the mentioned ones). They're really easy to use if you understand backoff algorithm itself. And as the backoff parameters could be stored in config, if backoff is failing too often, they could be adjusted to achieve better results.