NodeJs流管道是否对称？

Question

I'm building a server which transfers files from endpoint A to endpoint B.

I'm wondering if the NodeJs stream pipe is symmetric? If I do the following: request.get(A).pipe(request.put(B)); , does it upload as fast as it downloads?

I'm asking this question, because my server has an asymmetric connexion (it downloads faster than upload), and I try to avoid memory consumption.

Answer 1

According to node's documentation on stream#pipe pipe will switch the read stream to flowing mode - it will read only when the write stream has finished consuming previous packets.

readable.pipe() method attaches a Writable stream to the readable, causing it to switch automatically into flowing mode and push all of its data to the attached Writable. The flow of data will be automatically managed so that the destination Writable stream is not overwhelmed by a faster Readable stream.

So your transfer may be asymmetrical, due to different send/download speed - the difference may be buffered in Node's memory - Buffering of streams

Buffering#

Both Writable and Readable streams will store data in an internal buffer that can be retrieved using writable._writableState.getBuffer() or readable._readableState.buffer, respectively.

The amount of data potentially buffered depends on the highWaterMark option passed into the streams constructor. For normal streams, the highWaterMark option specifies a total number of bytes . For streams operating in object mode, the highWaterMark specifies a total number of objects.

Data is buffered in Readable streams when the implementation calls stream.push(chunk). If the consumer of the Stream does not call stream.read(), the data will sit in the internal queue until it is consumed.

Once the total size of the internal read buffer reaches the threshold specified by highWaterMark, the stream will temporarily stop reading data from the underlying resource until the data currently buffered can be consumed (that is, the stream will stop calling the internal readable._read() method that is used to fill the read buffer).

Data is buffered in Writable streams when the writable.write(chunk) method is called repeatedly. While the total size of the internal write buffer is below the threshold set by highWaterMark, calls to writable.write() will return true. Once the the size of the internal buffer reaches or exceeds the highWaterMark, false will be returned.

A key goal of the stream API, and in particular the stream.pipe() method, is to limit the buffering of data to acceptable levels such that sources and destinations of differing speeds will not overwhelm the available memory .

Because Duplex and Transform streams are both Readable and Writable, each maintain two separate internal buffers used for reading and writing, allowing each side to operate independently of the other while maintaining an appropriate and efficient flow of data. For example, net.Socket instances are Duplex streams whose Readable side allows consumption of data received from the socket and whose Writable side allows writing data to the socket. Because data may be written to the socket at a faster or slower rate than data is received, it is important each side operate (and buffer) independently of the other.

I recommend that you look at this question here the topic is elaborated a further.

If you run the following sample

const http = require('http');

http.request({method:'GET', host:'somehost.com', path: '/cat-picture.jpg'}, (response)=>{
  console.log(response);
}).end()

you can explore the underlying sockets - on my system they all have the highWaterMark : 16384 property. So if I understand the documentation, and the above-mentioned questions, in your case about 16KB may be buffered in the faster GET socket on Node.js level - what happens below is probably highly dependent on your system/network configuration.