[英]NodeJS: invalid array length Allocation failed - JavaScript heap out of memory
I'm trying to export data from a file and add this data to an Excel file using ExcelJS.我正在尝试从文件中导出数据并使用 ExcelJS 将此数据添加到 Excel 文件中。
worksheet.addRows(freedomRawData.records);
'records' is an array that contains more than 165,000 elements in it. 'records' 是一个包含超过 165,000 个元素的数组。 While writing the data to excel file, I get the error -
在将数据写入 excel 文件时,出现错误 -
FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
致命错误:无效的数组长度分配失败 - JavaScript 堆内存不足
The same script works for data with 'record' length 115,000 elements.相同的脚本适用于“记录”长度为 115,000 个元素的数据。 Over the internet, I found the below method:
通过互联网,我找到了以下方法:
node --max-old-space-size=8192 <file_name>.js
This didn't work and I tried using the maximum capacity of my RAM (16384MB);这不起作用,我尝试使用 RAM 的最大容量(16384MB); which didn't help as well.
这也没有帮助。 I'm a newbie in NodeJS, and any help would be much appreciated.
我是 NodeJS 的新手,任何帮助将不胜感激。
ExcelJS has a streaming writer interface: ExcelJS 有一个流式写入器接口:
async function writeExcel(rows_array){
const workbook = new Excel.stream.xlsx.WorkbookWriter({ filename: 'stream.xlsx' })
const worksheet = workbook.addWorksheet('data')
for (const row of rows_array) {
worksheet.addRow(row).commit()
}
await workbook.commit()
}
In this case, the data set could still present a memory problem due to the way the node/v8 garbage collector works.在这种情况下,由于 node/v8 垃圾收集器的工作方式,数据集仍然可能存在内存问题。 Garbage collection requires a tick of the event loop to actually clean up "freed" memory.
垃圾收集需要事件循环的滴答声来实际清理“释放”的内存。 If you run enough synchronous code, like a
for
loop, and that loop allocates more memory each iteration, then the GC can't run until after a pause in the code (ie the final worbook.commit()
. If you still run into OOM issues then you can force an async pause to allow the row memory that can be cleaned up after a .commit()
to actually be collected.如果您运行足够多的同步代码,例如
for
循环,并且该循环每次迭代分配更多内存,那么 GC 将无法运行,直到代码暂停(即最终的worbook.commit()
。如果您仍然遇到OOM 问题然后您可以强制异步暂停以允许实际收集.commit()
后可以清理的行内存。
for (const i in rows_array) {
worksheet.addRow(rows_array[i]).commit()
if (i % 10000 === 0) await Promise.resolve(true)
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.