NodeJS：达到堆限制分配失败 - memory 中的 JavaScript 堆

Question

我有一个带有 2 GB RAM 的 VPS (Debian)，我在其中运行pm2以执行 node.js 服务器应用程序。 我有一个缓存，每天晚上凌晨 3 点，我想将它转储到一个文件中作为备份。

每天凌晨 3 点，我都会收到以下错误：

<--- 最后几次 GC --->

[717119:0x63345b0] 66921944 ms：Mark-sweep 956.0 (1043.7) -> 954.9 (1043.7) MB，737.3 / 0.0 ms（平均 mu = 1.000，当前 mu = 0.349）分配失败； 清除可能不会成功 [717119:0x63345b0] 66922975 毫秒：标记扫描 962.8 (1043.7) -> 961.9 (1059.7) MB，978.0 / 0.0 毫秒（平均 mu = 0.999，当前 mu = 0.052）分配失败； 清除可能不会成功

<--- JS 堆栈跟踪 --->

致命错误：达到堆限制分配失败 - JavaScript 堆超出 memory 1：0xb6e500 node::Abort() [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 2：0xa7e632 [node / home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 3: 0xd47f20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [节点/home/userName /web/appName.com/public_html/appNameServer/appNameServer.js] 4: 0xd482c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [节点/home/userName /web/appName.com/public_html/appNameServer/appNameServer.js] 5: 0xf25685 [节点/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 6: 0xf37b6d v8::internal::Heap: :CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [节点/home/userName/web/appName.com/public_html/appNameServer/stoc> 7: 0xf1226e v8::internal ::HeapAllocator::AllocateRawWithLightRet rySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [节点/home/userName/web/app> 8: 0xf13637 v8::internal::HeapAllocator: :AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [节点/home/userName/web/stoc> 9: 0xef3b80 v8::internal::Factory ::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 10: 0xeeb5f4 v8::internal ::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [节点/home/userName/web /stoc>11: 0xeed988 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [节点/home/userName/web/appName.com/public_html /appNameServer/appNameServer.js] 12: 0x131ddec v8::internal::Inc rementalStringBuilder::Extend() [节点/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 13: 0x10349d8 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8 ::internal::String>) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 14: 0x1035c52 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier: :Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [节点/home/di>15: 0x10378af v8 ::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8:: internal::Object>) [node /home/di>16: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal: :Object>, bool, v8::internal::Handle<v8::internal::Object>) [节点/home/di>17: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier ：：连载 _(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>18: 0x10378af v8: :internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal ::对象>) [节点/home/di>19: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::对象>, bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>20: 0x10378af v8::internal::JsonStringifier::Result v8::internal::JsonStringifier: :Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [节点/home/di>21: 0x1037f13 v8 ::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>) [节点/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 22: 0x1039897 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier: :Serialize_(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [节点/home/d>23: 0x103ae5f v8 ::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node /hom>24: 0xdcdbf7 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node /home/userName/web/appName.com/public_html/appNameServer/appNameServer.js] 25: 0x16e9af9 [节点/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]

我知道错误很简单，简单的答案是添加更多 RAM，但我仍然想分享更多数据：

1. 在崩溃之前，应用程序消耗了 640mb（总容量），这意味着缓存本身消耗的比这个少。
2. 当我运行这个node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'结果是 1662。
3. 当我在本地运行缓存转储时，导出项目大约需要 3 秒，但在服务器上查看 pm2 日志时，尝试导出缓存项目到服务器再次备份需要 14 秒，好像导出需要很多时间更长的时间或者崩溃/重启可能需要那么多时间，不确定。

2022-12-27T03:00:00.016Z：导出到 json 2|appName|

2022-12-27T03:00:14.747Z：服务器正在端口 https 上运行...

根据上面的数字，640mb（毛）和节点至少还有 1gb 的事实，我不明白为什么一开始会有 memory 堆。

请指教，谢谢

Answer 1

NODE_OPTIONS（--max_old_space_size）

就我而言

$ node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
4144

但

$ export NODE_OPTIONS=--max_old_space_size=8192 && node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
8240

所以也许您可以使用此变量修改可用 memory 的数量。

使用缓冲区写入文件

可能您正在使用

fs.writeFileSync('/tmp/test.txt', content);

但也许你可以切换到

fs.createWriteStream(path, [options])

那么您就不必同时将文件的所有内容存储在 memory 中。

使用 IDE 进行分析

我使用这样的工具解决了类似的情况：

https://www.jetbrains.com/help/webstorm/v8-cpu-and-memory-profiling.html#ws_node_cpu_profiling

在控制台中，您可以简单地使用--porf标志启动脚本。 它将创建巨大的、难以阅读但非常详细的日志，讲述 memory 分配。