Apply several functions to each element in a stream

Question

I'm trying to take a stream elements, and apply a series of functions one element at a time. Each function one of several asynchronous operations (download, image transformations, uploads over a network, etc.). The libraries use different APIs (some Promises, some callbacks) so using Observables might simplify the API if I can figure out how to make this work.

Using rxjs I'm trying to work out the syntax to do this correctly, this is what I have attempted so far:

const Rx = require('rxjs');
const { download } = require('./lib/images');
const { transform } = require('./lib/operations');
const program = require('./lib/bin');
const util = require('./lib/util');

const files = util.fixFileExtensions(program.args);
const files$ = Rx.Observable.fromPromise(getMetadata(files));
const download$ = Rx.Observable.bindCallback(download);
const transform$ = Rx.Observable.fromPromise(transform);

// Take the files, apply the download$ and transform$ to each element in the stream
files$
    .map(download$)
    .map(transform$)
    .subscribe();

However my syntax is off and I get an error. What's the syntax to make this work?

EDIT From the current comments and answers, here's the updated but it's still not passing the output of download to transform . Here's what I have:

function download({id, name}, callback) {
    const credentials = {....};
    const filePath = `/path/to/tmp/${name}`;
    const stream = fs.createWriteStream(filePath);
    authenticate(credentials, err {
        if (err) throw err;
        files.download(...., (err, res) => {
            res.data
                .on('end', () => callback(filePath))
                .on('error', err => throw err)
                .on('data', /* write to file */)
                .pipe(stream)
        });
    });
}

function transform(filename, callback) {
     const transformations = /* load transformations */;
     const result = transformations.map(t => applyTransformation(filename, t));
     callback(result); 
}

const download$ = Rx.Observable.bindCallback(download);
const transform$ = Rx.Observable.bindCallback(transform);
files$
    .flatMap(file => file)
    .switchMap(download$)
    .map(transform$)
    .subscribe(console.log, console.error, () => console.log('map is done'));

This throws an error from my download function saying TypeError: callback is not a function , so download is called properly, but its output is not being passed to transform

Answer 1

Finally you need to take switchMap to apply asynchronous operations. Switchmap will subscribe to the inner Observable and resolve it:

files$
    .switchMap(download$)
    .switchMap(transform$)
    .subscribe();

If the value of download is needed in the subscription you must pass the inner Value:

files$
    .switchMap(download$)
    .switchMap(transform$, (outerValue, innerValue) => {outerValue, innerValue})
    .subscribe(valueOfDonwload => console.log(valueOfDonwload));

Answer 2

Leaving this here for posterity, David's reply is on the right track, but still doesn't work. I managed to make it work after perusing this blog post which shows how to work with Observable sequences: https://medium.freecodecamp.org/rxjs-and-node-8f4e0acebc7c

Here's what my final code looks like:

files$
    .flatMap(file => file)
    .switchMap(input => download$(input))
    .mergeMap(downloadedFile => transform$(downloadedFile))
    .subscribe(console.log, console.error, () => console.log('map is done'));

One thing I didn't expect is that my .switchMap(input => download$(input)) doesn't seem to be equivalent to .switchMap(download$) ; the latter syntax throws the error TypeError: callback is not a function

A second curious point is that .flatMap and .mergeMap are aliases - they have the same functionality which seems to boil down to "take the output of the previous Observable in the sequence". There's a nuance here that in my code, the first flatMap splits out the array from the previous Observable and the second one ( mergeMap ) takes the previous output and provides it as input to transform$ . Since the syntax isn't any different I'm not sure how that works unless it does this stuff by default

Now on to tying upload functionality to this sucker and continue to fully automate my side gig.