[英]aws sdk Multipart Upload to s3 with node.js
I am trying to upload large files to a s3 bucket using the node.js aws-sdk
.我正在尝试使用 node.js
aws-sdk
将大文件上传到 s3 存储桶。
the V2
method upload
integrally uploads the files in a multipart upload . V2
方式upload
整体上传文件在一个multipart upload 。
I want to use the new V3 aws-sdk.我想使用新的 V3 aws-sdk。 What is the way to upload large files in the new version?
新版本上传大文件的方式是什么? The method
PutObjectCommand
doesn't seem to be doing it. PutObjectCommand
方法似乎没有这样做。
I've seen there are methods such as CreateMultiPartUpload
but I can't seem to find a full working example using them.我已经看到有诸如
CreateMultiPartUpload
之类的方法,但我似乎找不到使用它们的完整工作示例。
Thanks in advance.提前致谢。
As of 2021, I would suggest using the lib-storage package, which abstracts a lot of the implementation details.截至 2021 年,我建议使用lib-storage package,它抽象了很多实现细节。
Sample code:示例代码:
import { Upload } from "@aws-sdk/lib-storage";
import { S3Client, S3 } from "@aws-sdk/client-s3";
const target = { Bucket, Key, Body };
try {
const parallelUploads3 = new Upload({
client: new S3({}) || new S3Client({}),
tags: [...], // optional tags
queueSize: 4, // optional concurrency configuration
partSize: 5MB, // optional size of each part
leavePartsOnError: false, // optional manually handle dropped parts
params: target,
});
parallelUploads3.on("httpUploadProgress", (progress) => {
console.log(progress);
});
await parallelUploads3.done();
} catch (e) {
console.log(e);
}
Source: https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/README.md来源: https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/README.md
Here's what I came up with, to upload a Buffer as a multipart upload, using aws-sdk v3 for nodejs and TypeScript.这是我想到的,使用 aws-sdk v3 for nodejs 和 TypeScript 将缓冲区上传为分段上传。
Error handling still needs some work (you might want to abort/retry in case of an error), but it should be a good starting point... I have tested this with XML files up to 15MB, and so far so good.错误处理仍然需要做一些工作(如果出现错误,您可能想中止/重试),但这应该是一个很好的起点……我已经用 XML 个最大 15MB 的文件对此进行了测试,到目前为止一切顺利。 No guarantees, though;
但是,没有任何保证; ;)
;)
import {
CompleteMultipartUploadCommand,
CompleteMultipartUploadCommandInput,
CreateMultipartUploadCommand,
CreateMultipartUploadCommandInput,
S3Client,
UploadPartCommand,
UploadPartCommandInput
} from '@aws-sdk/client-s3'
const client = new S3Client({ region: 'us-west-2' })
export const uploadMultiPartObject = async (file: Buffer, createParams: CreateMultipartUploadCommandInput): Promise<void> => {
try {
const createUploadResponse = await client.send(
new CreateMultipartUploadCommand(createParams)
)
const { Bucket, Key } = createParams
const { UploadId } = createUploadResponse
console.log('Upload initiated. Upload ID: ', UploadId)
// 5MB is the minimum part size
// Last part can be any size (no min.)
// Single part is treated as last part (no min.)
const partSize = (1024 * 1024) * 5 // 5MB
const fileSize = file.length
const numParts = Math.ceil(fileSize / partSize)
const uploadedParts = []
let remainingBytes = fileSize
for (let i = 1; i <= numParts; i ++) {
let startOfPart = fileSize - remainingBytes
let endOfPart = Math.min(partSize, startOfPart + remainingBytes)
if (i > 1) {
endOfPart = startOfPart + Math.min(partSize, remainingBytes)
startOfPart += 1
}
const uploadParams: UploadPartCommandInput = {
// add 1 to endOfPart due to slice end being non-inclusive
Body: file.slice(startOfPart, endOfPart + 1),
Bucket,
Key,
UploadId,
PartNumber: i
}
const uploadPartResponse = await client.send(new UploadPartCommand(uploadParams))
console.log(`Part #${i} uploaded. ETag: `, uploadPartResponse.ETag)
remainingBytes -= Math.min(partSize, remainingBytes)
// For each part upload, you must record the part number and the ETag value.
// You must include these values in the subsequent request to complete the multipart upload.
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
uploadedParts.push({ PartNumber: i, ETag: uploadPartResponse.ETag })
}
const completeParams: CompleteMultipartUploadCommandInput = {
Bucket,
Key,
UploadId,
MultipartUpload: {
Parts: uploadedParts
}
}
console.log('Completing upload...')
const completeData = await client.send(new CompleteMultipartUploadCommand(completeParams))
console.log('Upload complete: ', completeData.Key, '\n---')
} catch(e) {
throw e
}
}
I am trying to upload large files to a s3 bucket using the node.js aws-sdk
.我正在尝试使用 node.js
aws-sdk
将大文件上传到 s3 存储桶。
the V2
method upload
integrally uploads the files in a multipart upload . V2
方法upload
是在分段上传中整体上传文件。
I want to use the new V3 aws-sdk.我想使用新的 V3 aws-sdk。 What is the way to upload large files in the new version?
新版本上传大文件的方法是什么? The method
PutObjectCommand
doesn't seem to be doing it. PutObjectCommand
方法似乎没有这样做。
I've seen there are methods such as CreateMultiPartUpload
but I can't seem to find a full working example using them.我已经看到有诸如
CreateMultiPartUpload
之类的方法,但我似乎找不到使用它们的完整工作示例。
Thanks in advance.提前致谢。
Here is the fully working code with AWS SDK v3这是 AWS SDK v3 的完整工作代码
import { Upload } from "@aws-sdk/lib-storage";
import { S3Client, S3 } from "@aws-sdk/client-s3";
import { createReadStream } from 'fs';
const inputStream = createReadStream('clamav_db.zip');
const Bucket = process.env.DB_BUCKET
const Key = process.env.FILE_NAME
const Body = inputStream
const target = { Bucket, Key, Body};
try {
const parallelUploads3 = new Upload({
client: new S3Client({
region: process.env.AWS_REGION,
credentials: { accessKeyId: process.env.AWS_ACCESS_KEY, secretAccessKey: process.env.AWS_SECRET_KEY }
}),
queueSize: 4, // optional concurrency configuration
partSize: 5242880, // optional size of each part
leavePartsOnError: false, // optional manually handle dropped parts
params: target,
});
parallelUploads3.on("httpUploadProgress", (progress) => {
console.log(progress);
});
await parallelUploads3.done();
} catch (e) {
console.log(e);
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.