Node.js Promise.all - JavaScript 堆出 memory

Question

I tried to fetch some data in node.js in parallel, so the program is faster compared to fetching them sequentially. I do it like that:

function func() {
  var promises = [];
  for (let route of routes) {
    promises.push (calc(route, amount_wei));
  }
  Promise.all(promises).then(function(completedItems) {
    completedItems.forEach(function(val) {
      var result_amount = val;
      if (result_amount[5] > amount_start * 1) {
        console.log("Good Trade");
      }
    }
  });
}

I got help for this code here .

I do have like 10 routes, so the for loop should run 10 iterations. But no matter what I try, I got an error:

<--- Last few GCs --->

[23272:000001FDFC5C4DE0]   126463 ms: Mark-sweep (reduce) 9852.7 (10047.2) -> 9852.3 (10047.9) MB, 9776.7 / 0.0 ms  (average mu = 0.350, current mu = 0.000) allocation failure scavenge might not succeed
[23272:000001FDFC5C4DE0]   136795 ms: Mark-sweep (reduce) 9853.4 (10047.9) -> 9853.0 (10048.7) MB, 10314.7 / 0.0 ms  (average mu = 0.201, current mu = 0.002) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6487D7C4F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207
 2: 00007FF648765EC6 DSA_meth_get_flags+65542
 3: 00007FF648766D7D node::OnFatalError+301
 4: 00007FF64909B6CE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF649085CAD v8::SharedArrayBuffer::Externalize+781
 6: 00007FF648F2907C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF648F26194 v8::internal::Heap::CollectGarbage+4244
 8: 00007FF648F23B10 v8::internal::Heap::AllocateExternalBackingStore+2000
 9: 00007FF648F48696 v8::internal::Factory::NewFillerObject+214
10: 00007FF648C7AB65 v8::internal::DateCache::Weekday+1797
11: 00007FF6491293C1 v8::internal::SetupIsolateDelegate::SetupHeap+494417
12: 00007FF6490EB013 v8::internal::SetupIsolateDelegate::SetupHeap+239523
13: 000001FD8014DF37

I read nearly every page corresponding to this error but cannot solve it. I tried to set an environmental variable NODE_OPTIONS to --max-old-space-size=8192 . I tried to run the program with the command: node --max-old-space-size=8192 index.js

If it helps, I got my heapStatistics here:

{
  total_heap_size: 55885824,
  total_heap_size_executable: 786432,
  total_physical_size: 55885824,
  total_available_size: 10505880728,
  used_heap_size: 29100784,
  heap_size_limit: 10536091648,
  malloced_memory: 794688,
  peak_malloced_memory: 4109184,
  does_zap_garbage: 0,
  number_of_native_contexts: 1,
  number_of_detached_contexts: 0
}

How can I solve this issue?

Answer 1

Based on your original code in your previous question, the answer there missed one very important detail...

while(true) {
    func();
}

Also, the code in the answer won't even parse!!! But you must've figured that out for yourself or you wouldn't have the issue you are having (a missing ) in that answer)

Now that you are no longer waiting for the calc function "inline", that loop will runaway using as much resources until bang it dies from lack of memory or whatever

You need to wait for func to finish each "round" of calc s

So - perhaps something like

async function func() {
    const promises = routes.map(route => calc(route, amount_wei));
    const completedItems = await Promise.all(promises);
    completedItems.forEach(function (result_amount) {
        if (result_amount[5] > amount_start * 1) {
            console.log("Good Trade");
        }
    });
}

Now, you must

while (true) {
    await func();
}

Though, there's probably better way to do so, so that each route can work independently, re-running calc - but that's an exercise for you if you believe you want to do it that way