How to convert node readable stream to RX observable

Question

If I have a Node.js stream, say for example from something like process.stdin or from fs.createReadStream , how can I convert this to be an RxJs Observable stream using RxJs5?

I see that RxJs-Node has a fromReadableStream method, but that looks like it hasn't been updated in close to a year.

Answer 1

For anyone looking for this, following Mark's recommendation, I adapted rx-node fromStream implementation for rxjs5.

import { Observable } from 'rxjs';

// Adapted from https://github.com/Reactive-Extensions/rx-node/blob/87589c07be626c32c842bdafa782fca5924e749c/index.js#L52
export default function fromStream(stream, finishEventName = 'end', dataEventName = 'data') {
  stream.pause();

  return new Observable((observer) => {
    function dataHandler(data) {
      observer.next(data);
    }

    function errorHandler(err) {
      observer.error(err);
    }

    function endHandler() {
      observer.complete();
    }

    stream.addListener(dataEventName, dataHandler);
    stream.addListener('error', errorHandler);
    stream.addListener(finishEventName, endHandler);

    stream.resume();

    return () => {
      stream.removeListener(dataEventName, dataHandler);
      stream.removeListener('error', errorHandler);
      stream.removeListener(finishEventName, endHandler);
    };
  }).share();
}

Note that it intrinsically breaks all back pressure functionalities of streams. Observables' are a push technology. All input chunks are going to be read and pushed to the observer as quickly as possible. Depending on your case, it might not be the best solution.

Answer 2

The following should work for both v4 and v5 ( disclaimer untested):

fromStream: function (stream, finishEventName, dataEventName) {
    stream.pause();

    finishEventName || (finishEventName = 'end');
    dataEventName || (dataEventName = 'data');

    return Observable.create(function (observer) {

      // This is the "next" event
      const data$ = Observable.fromEvent(stream, dataEventName);

      // Map this into an error event
      const error$ = Observable.fromEvent(stream, 'error')
        .flatMap(err => Observable.throw(err));

      // Shut down the stream
      const complete$ = Observable.fromEvent(stream, finishEventName);

      // Put it all together and subscribe
      const sub = data$
        .merge(error$)
        .takeUntil(complete$)
        .subscribe(observer);

      // Start the underlying node stream
      stream.resume();

      // Return a handle to destroy the stream
      return sub;
    })

    // Avoid recreating the stream on duplicate subscriptions
    .share();
  },

Answer 3

The answers above will work, though the don't support backpressure. If you are trying to read a large file with createReadStream, they will read the whole thing in memory.

Here is my implementation with backpressure support: rxjs-stream

Answer 4

RxJs-Node 实现基于 RxJs4，但无需太多工作即可移植到 RxJs5 https://github.com/Reactive-Extensions/rx-node/blob/87589c07be626c32c842bdafa782fca5924e749c/index.js#L52

Answer 5

Since Node v11.14.0 streams support for await https://nodejs.org/api/stream.html#stream_readable_symbol_asynciterator

it means you can pass stream to from() operator.

Under hood rxjs(v7.xx) will call fromAsyncIterable() that will return Observable

Answer 6

Anyone coming here recently (RxJS 7 & Node 18+) should use the following code.

Why does this work? RxJS has been updated to handle stream like objects. When you pass a ReadStream to RxJS, it tests if it is ReadableStreamLike and then turns it into an AsyncGenerator .

import { from } from 'rxjs';

const file = fs.createReadStream(fileName);

const file$ = from(file).subscribe({
  next:  (dat) => { ... },
  error: (err) => { ... },
  complete: () => { ... }
});