How can I asynchronously handle stream events in Node.js while preserving order?

Question

I have a stream with some events:

    responseStream.on('end', resolve)
    responseStream.on('error', reject)
    responseStream.on('data', doStuff)

And

const doStuff = async (chunk) => {
  return await new Promise((resolve) => setTimeout(() => {
    console.log(chunk.toString())
    resolve()
  }, Math.random() * 1000))
}

What I want to do is keep the order that events are emitted in. But that doesn't happen since the handler takes different time. What's the best way to preserve the order of the emitted events?

Answer 1

You will need an async-aware queue or a promise chain in order to queue up events as they come in and hold them until the previous asynchronous event handlers are done. I started implementing it as a queue, but then decided that a promise chain is probably easier.

Here's an idea for how to do it:

class eventQueue {
    constructor(emitter, errorHandler) {
        this.emitter = emitter;
        this.errorHandler = errorHandler;
        this.chain = Promise.resolve();
        this.err = null;
        this.emitter.on('error', this.processError.bind(this));
    }
    processError(err) {
        // only ever call the errorHandler once
        if (this.err) return;
        this.err = err;
        this.errorHandler(err);
    }
    on(event, handler) {
        this.emitter.on(event, (...args) => {
            // wait for previous events to be done before running this one
            // and put the new end of the chain in this.chain
            this.chain = this.chain.then(() => {
                // skip any queued handlers once we've received an error
                if (this.err) return;
                // now that the chain has gotten to us, call our event  handler
                return handler(...args);
            }).catch(err => {
                this.processError(err);
                throw err;
            });
        });
        return this;
    }
}

Each incoming event is added onto a promise chain and won't get executed until all previous events have resolved any promise they returned.

Then, in your pseudo-code, you'd do something like this:

let queue = new eventQueue(responseStream, reject);

queue.on('end', resolve);
queue.on('data', doStuff);

const doStuff = (chunk) => {
  return new Promise((resolve) => setTimeout(() => {
    console.log(chunk.toString())
    resolve()
  }, Math.random() * 1000))
}

Note, the eventQueue, already has a built-in listener for the error event so you don't need to explicitly set one. It will call the errorHandler you pass to the constructor. Any promise that rejects will also cause that same errorHandler to be called. Once the errorHandler is called, no more event handlers will be called. The queue will be locked in an error state.