Can't access data stored from a node.js ReadStream?

Question

I've got this node.js file reading a .csv through a data stream like such:

shldata = []

fs.createReadStream('./spreadsheets/shl.csv')
.pipe(csv({separator:';'}))
.on('data', (row) => {
    shldata.push(row);
})
.on('end',() => {
    console.log('CSV file successfully processed');
});

console.log(shldata);

BTW I couldn't find another way to do this if not via a read stream.

If I console.log the data inside the ".on('end')" promise, it gives me the data. If I do it afterwards, it gives me null.

I get a feeling the stream gets queued and reads only after logging the data variable.

I've tried adding "return shldata" to the promise to no avail.

Also, I'm trying to delete some of the columns of the CSV that I won't need. I tried adding this for loop inside the promise:

for (i=0; i < shldata.length; i++){
    delete shldata[i]['key1'];
    delete shldata[i]['key2'];
    (...)
}

It works, but only if I repeat that line for each column header. If I try to use a key array like such, it doesn't work:

for (i=0; i < shldata.length; i++){
    delete shldata[i][keystoRemove];
}

Where the array "keystoRemove" is set prior to the stream. I got a feeling that I need to input this array somehow into this promise, otherwise it won't read it.

This all feels needlessly complicated. I'm just trying to write a script to read a few files, change their data a little bit and submit an HTTP post to an API.

Answer 1

You can access shldata in its entirety inside the 'end' event handler.

.on('end',() => {
    console.log('CSV file successfully processed');
    // at this point you have all the data.
}

Your current code is trying to access it before all the data's been processed and all the events have fired. You could, if this were a dedicated function, return shldata; inside this event handler.