node.js - 如何调整 child_process.spawn() 的标准输出缓冲区大小（又名 highWaterMark）？

Question

I'm attempting to adjust the buffer size (highWaterMark) for the stdout from a child_process.spawn() call, and I'm having a hard time doing this successfully. Below is some code that illustrates what I'm looking for. I'm trying to perform some benchmarking with various stdout buffer sizes, but until I can actually adjust the buffer size, I won't be able to do so. Any advice would be appreciated, thanks!

// I'm using the `stdout` from a child_process.spawn() call to feed into another stream, and I'm having a hard time increasing the buffer size on stdout.
// The code below illustrates the issue:
const child_process = require('child_process');
const stream = require('stream');
const fs = require('fs');

const compressStream = child_process.spawn("cat", ["testfile_32MB"], {writableHighWaterMark: 1024*1024, highWaterMark: 1024*1024});  // <-- child_process.spawn does not have highWaterMark or writableHighWaterMark options, so including them has no effect, but I'm including it just as an illustration of what I want.
compressStream.stdout.writableLength = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.writableLength = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.writableHighWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout.writableHighWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout._readableState.highWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.highWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout.on("data", (data) => {
  console.log("data length [from spawn()]:", data.length); // <-- data.length will always be <= 64KB! How can we get it higher?
});

const readStream = fs.createReadStream("testfile_32MB", {highWaterMark: 1024*1024});
readStream.on("data", (data) => {
  console.log("data length [from createReadStream()]:", data.length); // <-- This prints values <= 1MB, which is what I wish I had for child_process.spawn/stdout...
});

Note, you'll need a file called "testfile_32MB" in place to run this example code, and it should be at least a few megabytes in size.

Answer 1

Alas, you cannot do this. highWaterMark is a mechanism that is used to signal the writer to stop because the reader is not reading and its buffer is full.

A Socket will send you the data as soon as it is available. It will always use an internal hard-coded 64k buffer. Increasing the highWaterMark will make it capable of holding more data, but it will still send you 64Kb chunks - except that you will get bursts of several calls per event loop rotation.

You don't have access to the Stream creation options when using spawn - you can eventually pass a preconstructed named pipe stream - but this won't solve your problem.