[英]putObject() times out on files larger than ~1MB
當嘗試使用putObject()
函數將CSV文件上傳到我的S3存儲桶時,如果文件大於大約1 MB,則會超時。 我能夠成功上傳的最大CSV文件為1048 KB(25500行),在我嘗試上傳的6次中,只有4次成功。 任何小於1 MB的文件都可以快速上傳而不會失敗。 一旦功能達到約1 MB的閾值,會導致該功能掛斷並超時?
相關代碼如下:
function writeToS3(){ var params = { Bucket: bucketName, Key: fileName, Body: fs.createReadStream(fileName) } s3.putObject(params, function(err, data){ if(err){ console.log('Error uploading data to S3 bucket: ' + err); throw err; } else{ console.log("Data uploaded to S3 bucket: ", util.inspect(data)); } }); }
下面是錯誤消息:
Error uploading data to S3 bucket: TimeoutError: Connection timed out after 120000ms
C:\Users\426782\documents\workspace-sts\lf-rti-file-copier-sql\node_modules\aws-sdk-proxy\node_modules\aws-sdk\lib\request.js:31
throw err;
^
TimeoutError: Connection timed out after 120000ms
at ClientRequest.<anonymous> (C:\Users\426782\documents\workspace-sts\lf-rti-file-copier-sql\node_modules\aws-sdk-proxy\node_modules\aws-sdk\lib\http\node.js:83:34)
at ClientRequest.g (events.js:292:16)
at emitNone (events.js:86:13)
at ClientRequest.emit (events.js:185:7)
at Socket.emitTimeout (_http_client.js:630:10)
at Socket.g (events.js:292:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:338:8)
at ontimeout (timers.js:386:11)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)
npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "test" "65056"
npm ERR! node v6.11.4
npm ERR! npm v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! lf-rti-file-copier@1.0.0 test: `serverless invoke local -f dataRefresh -s dev -d "65056"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the lf-rti-file-copier@1.0.0 test script 'serverless invoke local -f dataRefresh -s dev -d "65056"'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the lf-rti-file-copier package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! serverless invoke local -f dataRefresh -s dev -d "65056"
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs lf-rti-file-copier
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls lf-rti-file-copier
npm ERR! There is likely additional logging output above.
我遇到了類似的問題,建議您嘗試使用此方法upload.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
如果有效載荷足夠大,則使用對零件的智能並發處理來上傳任意大小的緩沖區,blob或流。
這是一個示例,如何轉換代碼:
function writeToS3() {
fs.readFile(fileName, 'utf8', function (err, contents) {
if (err) {
console.log('Error during reading file: ' + err);
throw err;
} else {
// Create a binary data from stream
var base64data = Buffer.from(contents, 'binary')
var params = {
Bucket: bucketName,
Key: fileName,
Body: base64data
}
s3.upload(params, function (err, data) {
if (err) {
console.log('Error uploading data to S3 bucket: ' + err);
throw err;
} else {
console.log("Data uploaded to S3 bucket: ", util.inspect(data));
}
});
}
});
}
更新:
另外,請看一下Bluebird
庫方法 ,這將幫助您將所有方法轉換為Promise,以使其更具可讀性。
更改超時
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.