流上传时JavaScript堆内存不足错误

Question

I have this function for a streaming upload:

function doChunkInput() {
  console.log('Put Objects Chunk');
  let stream = new Readable({
    read() {}
  })

  for(i=0; i < 10000000000; i++) {
    stream.push(' data');
    stream.push(' more data');
    stream.push(' and more data');
  }

  // Pay attention to this
  // null indicates the end of the stream, so the `data` event will be fired
  stream.push(null)

  const params = {
    Bucket: bucket,
    Body: stream,
    Key: `sitemap.1.xml`,
  };
  return cos.upload(params).promise();
}

When I run it I get this error:

<--- Last few GCs --->

[39640:0x10263e000]    27425 ms: Mark-sweep 1394.1 (1426.8) -> 1394.1 (1421.8) MB, 3253.8 / 0.0 ms  (+ 12.3 ms in 367 steps since start of marking, biggest step 11.6 ms, walltime since start of marking 3644 ms) (average mu = 0.156, current mu = 0.119) all[39640:0x10263e000]    27442 ms: Scavenge 1395.3 (1421.8) -> 1395.4 (1423.8) MB, 8.6 / 0.0 ms  (average mu = 0.156, current mu = 0.119) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x274ce15be3d]
Security context: 0x320c0d71e6e1 <JSObject>
    1: write [0x320cf221e191] [buffer.js:~896] [pc=0x274ce61b0ca](this=0x320c48cc5689 <Uint8Array map = 0x320cf21519e1>,string=0x320cd6757f31 <String[10]:  more data>,offset=0x320c0d73d7b9 <String[4]: utf8>,length=0x320cb18826f1 <undefined>,encoding=0x320cb18826f1 <undefined>)
    2: arguments adaptor frame: 2->4
    3: from [0x320cf221dbc1] [buffer.js:~199] ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003b125 node::Abort() [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 2: 0x10003b32f node::OnFatalError(char const*, char const*) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 3: 0x1001a89a5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 4: 0x100573dc2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 5: 0x100576895 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 6: 0x10057273f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 7: 0x100570914 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 8: 0x10056f518 v8::internal::Heap::HandleGCRequest() [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
 9: 0x100524d48 v8::internal::StackGuard::HandleInterrupts() [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
10: 0x1007d45b1 v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/me/.nvm/versions/node/v10.14.2/bin/node]
11: 0x274ce15be3d 
12: 0x274ce61b0ca 
13: 0x274ce10a5c3 
Abort trap: 6

How do I resolve it?

Answer 1

You should listen to backpressure. When .push returns false it means that you try to add more data than the consumer consumes. In that case you should stop, and wait for a further call to _read :

 let i = 0;
 let stream = new Readable({
    read() {
      while(i < 1000000000) {
         if(!this.push("data")) return;
         i++
      }
      this.push(null);
    }
 })