使用 nodejs SDK v3 将 stream 上传到 Amazon s3

Question

I would like to upload a stream to s3 using the AWS SDK v3.

The PutObjectRequest as documented here should accept a ReadableStream, but every time I receive the error NotImplemented: A header you provided implies functionality that is not implemented - the header reported is Transfer-Encoding .

This SO article suggests that I'm providing an empty body, but it's supposed to accept ReadableStream (Archiver provides a Transform stream).

I have created a minimal (non) working example. I have also tried putting client.send() with a callback instead, but with the same response.

import fs from 'fs'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import archiver from 'archiver'

async function run() {
  const archive = archiver('zip')

  const client = new S3Client({ region: process.env.AWS_REGION })

  // create a stream destination for aws
  // also pipe to aws
  try {
    const command = new PutObjectCommand({
      Bucket: process.env.AWS_DEFAULT_BUCKET,
      Key: 'testupload.zip',
      // set the content type as we're streaming it
      ContentType: 'application/zip',
      // body should take a readable stream
      Body: archive,
    })
    client.send(command, (err, data) => {
      console.log(data)
    })
  } catch (err) {
    console.error('Error uploading package to S3')
    console.error(err)
    res.status(501).json({ message: 'Error configuring upload to s3' })
  }

  archive.file('/fixtures/harambe.jpg', { name: 'harambe.jpg' })

  archive.finalize()
}

run()

Answer 1

The issue is that the Archiver library uses a static version of readable-stream from Node 10, which is not compatible with the AWS SDK version.

When it pipes to the Archiver instance, it fails.