Stream data in Node without sending content-length header

Question

I have this function which streams buffered data:

function doChunkInput() {
  console.log('Put Objects Chunk');
  let stream = new Readable({
    read() {}
  })

  for(i=0; i < 1000; i++) {
    stream.push(' data');
    stream.push(' more data');
    stream.push(' and more data');
  }

  // Pay attention to this
  // null indicates the end of the stream, so the `data` event will be fired
  stream.push(null)

  const params = {
    Bucket: bucket,
    Body: stream,
    Key: `sitemap.1.xml`,
  };
  return s3.upload(params).promise();

Is it possible to from a stream without buffering. What I want is to stream data where the content-length can't be calculted. In the above example it is getting calculated in the buffer.

I'm trying to test the chunked upload but with the above example the content-length header is added to the request and the transfer-encoding header is not.

Answer 1

Short answer is no. AWS SDK does not support streaming, but there's a nice way around it.

You can upload in parts of any size (mind you, as per anything AWS, you pay a tiny bit for each upload - lots of parts may grow to a visible cost). There's even a nice npm module called: s3-stream-upload which exposes a normal Writable stream interface that you can pipe to.

You could use my scramjet if you'd prefer to have some special logic around the upload - it's actually quite simple:

const { StringStream } = require("scramjet");

const params = {Key: 'filename', Bucket: 'my-bucket'};
const { UploadId } = await s3.createMultipartUpload(params).promise();
let i = 1;

StringStream
    .from(function*() {
        for(i=0; i < 1000; i++) {
           yield ' data';
           yield ' more data';
           yield ' and more data';
        }
    })
    .toBufferStream()
    .breakup(1048576)   // you'd buffer every 1meg
    .map(
        (Body) => s3.uploadPart({
            ...params,
            Body,
            UploadId,
            PartNumber: i++
        }).promise()
    )
    .toArray()
    .then(
        parts => s3.completeMultipartUpload({
            ...params,
            MultipartUpload: {
                Parts: parts.map(
                    ({ ETag }, i) => ({ ETag, PartNumber: i + 1 })
                )
            }
        }).promise()
    )
    .catch(
        e => {
            console.error("error, aborting", e);
            return s3.abortMultipartUpload({...params, UploadId})
        }
    )

I'm sorry - I didn't run the example above, so it's more of a guide what should happen - but I can fix up tomorrow if you get stuck.

---

edit: Now I understand that you want to know how to create a stream and add to it without building the whole stream upfront

I changed the answer above to create a stream from generator. You could also implement the read method in your example to return the 'data' via this.push ( see docs here ), but I guess it's way simpler with scramjet and generators.