归档器与 PassThrough()、append 组合/管道并最终确定不是 function - Nodejs Stream

Question

So, I am downloading files with the help of Axios stream, zipping them with archiver then I want to upload the zip to s3 bucket. At first, I saved zip in local directory, everything worked fine like this. I used multipipe lib to combine streams. Combine archiver zip stream with fs.createWriteStream

 function uploadFromStream(name) {
   const output = fs.createWriteStream(__dirname + `/${name}.zip`);

   const zip= combine(archive, output);
    return zip
 }

At the axios end

zip.append(response.data.pipe(new PassThrough()), { name: name });

Everything worked as expected, Zip is getting saved with all files in it.

Then I researched and look for how to upload to s3 with stream. I found this way. Pipe a stream to s3.upload()

inputStream.pipe(uploadFromStream(s3));

function uploadFromStream(s3) {
  var pass = new stream.PassThrough();

  var params = {Bucket: BUCKET, Key: KEY, Body: pass};
  s3.upload(params, function(err, data) {
    console.log(err, data);
  });

  return pass;
}

I implemented this with archiver like this.

function uploadFromStream(bucket, destination) {
  var pass = new stream.PassThrough();

  var params = { Bucket: bucket, Key: destination, Body: pass };
  S3.upload(params, function (err, data) {
    console.log(err, data);
  });

  const s3Stream = combine(archive, pass);

  return s3Stream;
}

Now when I try to append I am getting error append not a function same when finalize is not a function. When I check I found that s3Stream.append and s3Stream.finalize are undefined. I dont think It should happen but I guess PassThrough is removing functions from combined stream.

And piping streams like this also did not work.

  return archive.pipe(pass);

Answer 1

So after logging streams many times. I think pipe does not combine functions of two streams, but only creates a flow between two streams. So I just need to pipe passThrough and archiver then return archiver to append files to it.

function uploadFromStream(bucket, destination) {
  const pass = new stream.PassThrough();

  const params = { Bucket: bucket, Key: destination, Body: archive };

  S3.upload(params, function (err, data) {
    console.log(err, data);
  });
  archive.pipe(pass);

  return archive;
}

By doing this I am not facing the above mentioned error. My bad was working with streams first time