ENOMEM in Node.js caused by high virtual memory usage

Question

I've been getting ENOMEM errors in my production Node.js (v12) app for a couple of months.

This issue has been particularly hard to diagnose because my heap is always in the 100MB range and doesn't increase over time. Hence a node memory leak is unlikely.

However, some times the VIRT memory usage (as reported by top cmd) goes as high as 15-20GB and that's when node starts to throw some "spawn: enomem" errors.

As far as I see something is making node reserve lots and lots of virtual memory even though the heap is stable at 100MB. And I can't diagnose what it is since the node debuting tools only allow me to inspect the heap.

I've spent close to 100hrs on this problem so any insight / help would truly be appreciated !

I tried:

• Manualy setting --max-old-space-size
• Increasing RAM on the server
• Adding SWAP to my server and allowing docker to use that SWAP
• Inspecting the node heap with chrome dev tools
• Setting a memory limit for my docker containers
• Moving to a different process manager (pm2 instead of supervisord)
• upgrading all my packages and using the latest LTS version of node
• Reading and understanding all the google results for node ENOMEM
• Understanding how Node manages its memory pool, especially regarding heap and GC.
• Diving into linux kernel concepts of virtual, reserved, swap and shared memory.

Thanks for your help !

Answer 1

We managed to solve the problem. Keep in mind the solution might is specific to our own project.

We manage to reproduce the error consistently when using the package "imagemin" with "imagemin-jpegtran" & "imagemin-pngquant" on a large array (20k+) of files, and waiting for all promises to finish with a promise.all(). The VIRT mem usage surged to 20GB+ and was never freed after execution.

The solution for us, was to use plimit to limit the number of concurrent promises to 5 or 10. On the same number of images (20k+) the VIRT mem usage never rose, neither did RES, and we stopped getting ENOMEM errors.

Hope this helps someone out there