NodeJS - Wait Until Streaming Multiple Files Is Complete Before Continuing Code

Question

I'm new to Javascript and NodeJS. Im trying to read multiple CSV files before doing some processing on them. My current issue is when I run the code it tries to execute the processing before the reading of the file is complete. I would like to load both the CSVs before I start doing any processing on them.

Could some explain why this happens and how I can solve the problem in Javascript/NodeJS.

function readCSV(path){

    var events = []
    
    fs.createReadStream(path).pipe(csv()).on('data', (row) => {
        events.push(row);
    }).on('end', () => {
        console.log('CSV file successfully processed. Length: ' + events.length);
    }); 

    return events
}

function app(){

    var file_list = listFiles(folder_path);

    for (let i = 0; i < file_list.length; i++) {
        const file = file_list[i];
        var events = readCSV(file)
    }
    
    processCSV(events) // Some processing
}

app();

Any help would be great and any explanation on how I can control when code is executed would be much appreciated.

Answer 1

Sorry, your code cannot be compiled, so I can answer only with untested code.

My current issue is when I run the code it tries to execute the processing before the reading of the file is complete.

The main problem is that fs.createReadStream doesn't read the file, it requests the file system to start reading and calls your callbacks when some chunks were read, so event 'end' will be called much later, after readCSV completed and returned an empty result.

Your code was written as if you expect an synchronous answer, and you can make it work correctly with the use of sync methods like fs.readFileSync .

How to fix it in asynchronous way? Write CSV processing in "on(end)" callback or use promises.

Promises are much simpler and linear.

First make readCSV to return Promise .

function readCSV(path: string){ //return Promise<any[]> 
  return new Promise((resolve) => {
    var events = [];
    fs.createReadStream(path)
      .pipe(csv())
      .on('data', (row) => {
        // this code called in future
        events.push(row);
    }).on('end', () => {
        // this code called in future to, 
        console.log('CSV file successfully processed. Length: ' + events.length);
        resolve(events); //return csv parsed result
    }); 
  })
}

Then in main app, use Promise.all to wait all fileReading promises.

function app(){
  // i don't know what is listFiles,
  // i hope it returns sync result
  var file_list = fs.listFiles(folder_path);

  const dataPromises: Promise[] = []
  for (let i = 0; i < file_list.length; i++) {
      const file = file_list[i];
      //launch reading
      dataPromises.push(readCSV(file))
  }
  Promise.all(dataPromises).then(result => {
    //this code will be called in future after all readCSV Promises call resolve(..)
    for(const events of result){
      processCSV(events);    
    }
  })
}