How can I improve the speed of this code with proper async/await/promises

Question

Using Node.js, I have got a task to improve the code I have created. This code do 60 HTTP requests and uses libraries for that.

It takes 30 seconds to do all HTTP requests and save each to a file!

It is said that is possible to do those requests in 3 seconds with:

1. Proper management of async promises

2. A bit smarter caching

3. Not using cluster

4. Only add the overhead once

I am afraid that I am not sure where to start understand what I can do exactly.

So below code gets an array of 60 items where each one is one HTTP request:

 const exchanges = ccxt.exchanges 

Those goes into the: worker = async function and at the end of code: await Promise.all(workers) wait for them to finish.

I am not sure where to begin to actually be able to come down to 3 seconds. How can it be possible to improve the speed of this code?

 'use strict'; const ccxt = require ('ccxt') , log = require ('ololog').noLocate // npm install ololog , fs = require ('fs') // the numWorkers constant defines the number of concurrent workers // those aren't really threads in terms of the async environment // set this to the number of cores in your CPU * 2 // or play with this number to find a setting that works best for you , numWorkers = 8 ;(async () => { // make an array of all exchanges const exchanges = ccxt.exchanges .filter (id => ![ 'cap1', 'cap2' ].includes (id)) // instantiate each exchange and save it to the exchanges list .map (id => new ccxt[id] ({ 'enableRateLimit': true, })) // the worker function for each "async thread" const worker = async function () { // while the array of all exchanges is not empty while (exchanges.length > 0) { // pop one exchange from the array const exchange = exchanges.pop() // check if it has the necessary method implemented if (exchange.has['fetchTickers']) { // try to do "the work" and handle errors if any try { // fetch the response for all tickers from the exchange const tickers = await exchange.fetchTickers() // make a filename from exchange id const filename = '/myproject/tickers/' + exchange.id + 'Tickers.json' // save the response to a file fs.writeFileSync(filename, JSON.stringify({ tickers })); } catch (e) { } //Error } } } // create numWorkers "threads" (they aren't really threads) const workers = [ ... Array (numWorkers) ].map (_ => worker ()) // wait for all of them to execute or fail await Promise.all (workers) }) () 

Answer 1

I think you're making things more complicated than they need to be. You can do all of the work within the map callback, then use Promise.all(promises) to wait for all of the operations to complete. This process does take longer than the expected "3 seconds" (15 seconds in my case) and there are a lot of errors produced (like missing apiToken, or fetchTickers not being implemented), but this could be an issue with my environment (I've never used ccxt before and I don't have any apiTokens).

This is the implementation I came up with, hopefully it helps you meet your needs:

const ccxt = require('ccxt');
const fs = require('fs');
const path = require('path');

(async () => {
    const start = Date.now();

    const dumps = ccxt.exchanges
        .filter((id) => !['coinmarketcap', 'theocean'].includes(id))
        .map(async (id) => {
            const Exchange = ccxt[id];
            const exchange = new Exchange({enableRateLimit: true});
            if (exchange.has['fetchTickers']) {
                try {
                    const tickers = await exchange.fetchTickers();
                    const dumpFile = path.join(__dirname, 'exchanges', `${id}-Tickers.json`);
                    await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
                } catch (e) {
                    console.error(e);
                }
            }
        });

    await Promise.all(dumps);

    const end = Date.now();
    console.log(`Done in ${(end - start) / 1000} seconds`);
})();

Answer 2

I try to see if it would be possible to do it even faster. I try to cache up all possible memories needed.. Before actually doing the .fetchTickers() request.

I managed to go down even to 9 seconds from 15 seconds as it seemed. But this code below is even one step further but I do receive compile errors and are not sure what I am doing wrong.

Error is:

ReferenceError: id is not defined

Isn't id passed along in the 'exchange' object that is .pushed into 'exchangesArray'?

I am trying to put the exchange object into an array first with all the time that takes:

 var exchangesArray = []; 

Then with this "exchangesArray" in place, I try to execute the function that does the fetchTickers:

 'use strict'; const ccxt = require('ccxt'); const fs = require('fs'); const path = require('path'); //Cache some memories first var exchangesArray = []; (async () => { const allexchanges = ccxt.exchanges.filter((id) => !['coinmarketcap', 'theocean'].includes(id)) .map(async (id) => { const Exchange = ccxt[id]; const exchange = new Exchange({ enableRateLimit: true }); if (exchange.has['fetchTickers']) { exchangesArray.push(exchange); } }); await Promise.all(allexchanges); })(); //Use cached memories to do the "fetchTickers()" as fast as possible (async () => { const start = Date.now(); const exchanges = exchangesArray; while (exchanges.length > 0) { // pop one exchange from the array const exchange = exchanges.pop() try { const tickers = await exchange.fetchTickers(); const dumpFile = path.join(__dirname, 'exchanges', `${id}-Tickers.json`); await fs.promises.writeFile(dumpFile, JSON.stringify(tickers)); } catch (e) { console.error(e); } } await Promise.all(exchanges); const end = Date.now(); console.log(`Done in ${(end - start) / 1000} seconds`); })();