简体   繁体   中英

Memory leak using node, fs and readline

I'm trying to pass a huge csv data to sql and at half transaction it fails by memory leak: the error is exactly 'JavaScript heap out of memory'. Can you see anything that smells bad? Thank you in advance.

const readline = require('readline');
const fs = require('fs');

const postFromSarenetCSV = async (rq, res = response) => {
 try {
 const rl = readline.createInterface({
 input: fs.createReadStream('data/unparsedSarenet.csv',"UTF8"),
 crlfDelay: Infinity
 });
 await rl.on('line', async (line) => {
 //create object with papaparse library:
 let object = "column1,column2,column3\r\n"
 object = object.concat(line)
 let json = Papa.parse(object, {newline: "\r\n", header: true})
 //create sql request:
 var data = json.data[0]
 let {query, values} = createReplaceQuery('services', Object.entries(data))
 await dbConnection.query(query, values);
 });
 await events.once(rl, 'close');
 res.status(200).json({
 ok: true,
 msg: 'Database updated'
 })
 } catch (err) {
 console.error(err);
 }
} 

I've tried to use await at readline listener thinking that may the accumulation of readed lines create the memory leak, but do nothing to solve the problem. When i did logs to see the last one, the last one has been before papaparse.

I've found the problem. That's the listener. The listener doesn't wait to finalize his function. I've changed this line:

rl.on('line', async (line) => {

to this:

for await (const line of rl) {

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM