在Node.js / Gulp中重用流管道/子管道

Question

starting out with streams/gulp/vinyl and trying to clean up/DRY my gulpfile using stream-combiner2 . so far it is working well except now i need to run a sub-pipeline inside another pipeline (simplified psuedocode-ish):

var logger = require("gulp-logger"),
HTMLPrettify = require("gulp-html-prettify"),
combiner = require("stream-combiner2"),
through = require("through2"),
tap = require("gulp-tap"),
_ = require("lodash");


preprocessPipeline = function() {
  return combiner.obj(getFrontMatter(), processTemplates());
};


gulp.task("build", ["clean"], function() {
    return gulp.src("in/**/*.html")
        .pipe(preprocessPipeline())
        .pipe(tap(function(file) {
            var data, dataSources;
            dataSources = getDataSources(file);
            data = {};

            /* PAUSE MAIN PIPELINE HERE */
            /* RUN THIS SUB PIPELINE BELOW */
            gulp.src(dataSources)
                .pipe(preprocessPipeline())
                .pipe(tap(function(file) {
                    _.merge(data, file.data);
                }));

            file.data = data;
            /* RESUME MAIN PIPELINE */
        }))
        .pipe(HTMLPrettify())
        .pipe(gulp.dest("out"));
});

as you can see, inside the main "build" pipeline i am trying to do the following for each vinyl file object that comes down the main pipeline:

• pause execution of main "build" pipeline
• get the data source paths for the current vinyl file via the synchronous getDataSources() function
• use another sub pipeline to process those data source paths, reusing the preProccessPipeline and eventually merging their data
• finally the merged data from the data sources is added the the vinyl file in the main pipeline and execution continues in the main pipeline

gulp.src() looks like the perfect way to load these data source files into the sub-pipeline but the main pipeline seems to be completing before the sub-pipeline starts.

also, i am starting to realize that perhaps my pipelines are looking like synchronous function flow and most likely i am missing a key piece of the stream/node.js puzzle.

related docs that have helped me reuse pipelines but not with sub-pipelines:

• https://github.com/gulpjs/gulp/blob/master/docs/recipes/sharing-streams-with-stream-factories.md
• https://github.com/gulpjs/gulp/blob/master/docs/recipes/make-stream-from-buffer.md

related issues:

• is there any way to reuse a chain of pipe transformations in NodeJS?

thanks.

Answer 1

Seems like you're using gulp as a templating engine of sorts—totally doable.

Gulp lets you specify which tasks are dependent on others, so you could break out that sub-pipeline into a new task, say "buildDataFiles" , then have your build task depend on that one ie

gulp.task("build", ["buildDataFiles", "clean"], function(){

But the problem with this approach is, as you said, "it is not possible to know the data file(s) associated with a template file until the template is processed." If you're sure there's no way around that, it gets trickier. What if instead you broke out the main pipe into a "processTemplate" task, the "build" task then only handles the sub-pipeline but depends on "processTemplate" ? It could look something like this:

gulp.task("processTemplate", ["clean"], function() {
  gulp.src("in/**/*.html")
    .pipe(preprocessPipeline())
    .pipe(gulp.dest("build/temp"));
}

gulp.task("build", ["processTemplate"], function() {
  gulp.src("build/temp/*.html")
    .pipe(tap(function(file) {
      var data, dataSources;
      dataSources = getDataSources(file);
      data = {};

      gulp.src(dataSources)
        .pipe(preprocessPipeline())
        .pipe(tap(function(file) {
          _.merge(data, file.data);
        }));

      file.data = data;

    }))
  .pipe(HTMLPrettify())
  .pipe(gulp.dest("out"));
}

It looks really similar, but the problem your running into requires that the templates be processed first, then the data files. I would try checking out gulp-consolidate , it may help you out.

Answer 2

I find stream-combiner much more straight to the point

function coffeePipe() {
  return Combine(
    coffeescript(),
    coffeelint.reporter('fail')
  )
}
//usage:
gulp.src().pipe(coffeePipe());