GDAX API - Rate limit exceeded

Question

I get an error message saying "Rate limit exceeded" when I try to request historical data from GDAX. I use promises and setInterval to request historical price data from GDAX like this:

let promiseArray = //Array with promises
let j = 0;
let grabPrices = setInterval(() => {
    if (j === promiseArray.length) {
        //Do something when all promises have been resolved
    } 
    else {
        promiseArray[j].then(
            data => {
                //Work with data
            }
        ).catch(error => console.log('An error occurred: ' + error));
        j++;
    }
}, 1000) //I have even tried to use 10 seconds (10000) as a second argument to setInterval, but it doesn't make any difference.

FROM the official API documentation

Rate Limits When a rate limit is exceeded, a status of 429 Too Many Requests will be returned.

REST API PUBLIC ENDPOINTS We throttle public endpoints by IP: 3 requests per second, up to 6 requests per second in bursts.

Answer 1

When you have a promise then the request is already made so your promiseArray is an array of ongoing requests.

Let's say I have an array of urls and use fetch to get the content. Using map to map the url to a promise and give the array of promises to Promise.all :

Promise.all(
  urls.map(fetch)
).then(
  resulst=>...
);

If urls has 10 items this program will make 10 requests immediately.

You can pass the fetch function to a throttle function that will schedule fetch in such a way that it will only be called 3 times a second. The throttle function will return a promise but will not call fetch immediately.

The throttlePeriod function can be found here . If you're only going to copy paste code from that module and not import the whole module then you need the later function as well because throttlePeriod depends on it.

const fetchMaxThreePerSecond = throttlePeriod(3,1100)/*give it an extra 100*/(fetch)
Promise.all(
  urls.map(fetchMaxThreePerSecond)
).then(
  resulst=>...
);

That takes care of the throttling but if you know how Promise.all works you know that all promises reject if only one rejects. So if you have a hundred urls and 99 resolve but one rejects your .then never gets called. You will loose the 99 successful requests.

You could pass a promise to Promise.all that will not reject, you can do so by catching a rejected promise and in the catch return a special value that you can filter out when processing results:

const fetchMaxThreePerSecond = throttlePeriod(3,1100)/*give it an extra 100*/(fetch);
const Fail = function(reason){this.reason = reason;};
const isFail = x=>(x&&x.constructor)===Fail;
const isNotFail = x=>!isFail(x);
Promise.all(
  urls.map(
    url=>
      fetchMaxThreePerSecond(url)
      .catch(e=>new Fail([url,e]))//save url and error in Fail object
  )
)
.then(//even if a request fails, it will not reject
  results=>{
    const successes = results.filter(isNotFail);
    const failed = results.filter(isFail);
    if(failed.length!==0){
      console.log("some requests failed:");
      failed.forEach(
        (fail)=>{
          const [url,error] = fail.reason;
          console.log(`Url: ${url}`);
          console.log(JSON.stringify(error,undefined,2));
        }
      )
    }
  }
); 

Answer 2

I have no problem with GDAX history calls when I set my rate limiter to 3 API calls every 2.1 seconds.

I have some occasional problems when I set my rate limiter to 3 API calls every 1.8 seconds.

I have tested these values with automated tests.

Important: all my calls to GDAX share the same rate limiter(!)

To be save, use the code from @HMR's answer together with my tested parameters.