buffer and stream - how are they related?

Question

I am putting some code here:

const { createReadStream, ReadStream } = require('fs');

var readStream = createReadStream('./data.txt');

readStream.on('data', chunk => {
  console.log('---------------------------------');
  console.log(chunk);
  console.log('---------------------------------');
});

readStream.on('open', () => {
  console.log('Stream opened...');
});

readStream.on('end', () => {
  console.log('Stream Closed...');
});

So, stream is the movement of data from one place to another. In this case, from data.txt file to my eyes since i have to read it.

I've read in google something like this:

Typically, the movement of data is usually with the intention to process it, or read it, and make decisions based on it. But there is a minimum and a maximum amount of data a process could take over time. So if the rate the data arrives is faster than the rate the process consumes the data, the excess data need to wait somewhere for its turn to be processed.

On the other hand, if the process is consuming the data faster than it arrives, the few data that arrive earlier need to wait for a certain amount of data to arrive before being sent out for processing.

My question is: which line of code is "consuming the data, processing the data"? is it console.log(chunk) ? if I had a huge time consuming line of code instead of console.log(chunk) , how would my code not grab more data from buffer and wait until my processing is done? in the above code, it seems like, it would still come into readStream.on('data')'s callback..

Answer 1

My question is: which line of code is "consuming the data, processing the data"

The readStream.on('data', ...) event handler is the code that "consumes" or "processes" the data.

if I had a huge time consuming line of code instead of console.log(chunk), how would my code not grab more data from buffer and wait until my processing is done?

If the time consuming code is synchronous (eg blocking), then no more data events can happen until after your synchronous code is done because only your event handler is running (in the single-threaded event loop driven architecture of node.js). No more data events will be generated until you return control back from your event handler callback function.

If the time consuming code is asynchronous (eg non-blocking and thus has returned control back to the event loop), then more data events certainly can happen even though a prior data event handler has not entirely finished it's asynchronous work yet. It is sometimes appropriate to call readStream.pause() while doing asynchronous work to tell the readStream not to generate any more data events until you are ready for them and you can then readStream.resume() .