[英]Uploading large size (more than 1 GB) file on Amazon S3 using Java: Large files temporary consuming lot of space in server
I am trying to upload large files (more than 1 GB) on amazon S3 using Java我正在尝试使用 Java 在 amazon S3 上上传大文件(超过 1 GB)
I am using AWS S3 multipart upload to upload large files in chunks.我正在使用 AWS S3 分段上传来分块上传大文件。 https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileJava.html
https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileJava.html
I am using also uploading the files in chunks from the frontend.我也在使用从前端分块上传文件。
So, the file being uploaded will be temporarily uploaded on the server in chunks and it will be uploaded on S3 in chunks.因此,正在上传的文件将暂时分块上传到服务器上,并将分块上传到 S3。 Now the problem is that this method puts a huge load on the server since this consumes server space temporarily.
现在的问题是这种方法会给服务器带来巨大的负载,因为这会暂时占用服务器空间。 If multiple users are trying to upload large files at the same time then it will create an issue.
如果多个用户同时尝试上传大文件,则会产生问题。
Is there any way of directly uploaded files from the user's system to amazon S3 in chunks without storing the file on server temporarily?有没有办法将文件从用户系统直接上传到亚马逊 S3,而无需将文件临时存储在服务器上?
If upload the files via frontend directly then there a major risk of keys getting exposed.如果直接通过前端上传文件,则存在密钥暴露的重大风险。
You should leverage the upload directly from client with Signed URL There are plenty documentation for this您应该直接从带有签名 URL 的客户端利用上传有很多相关文档
AWS SDK Presigned URL + Multipart upload AWS SDK 预签名 URL + 分段上传
The presigned URLs are useful if you want your user/customer to be able to upload a specific object to your bucket , but you don't require them to have AWS security credentials or permissions. 如果您希望您的用户/客户能够将特定对象上传到您的存储桶,但不要求他们拥有 AWS 安全凭证或权限,则预签名 URL 非常有用。
You could also be interested in limiting the size that user is able to upload您也可能对限制用户能够上传的大小感兴趣
Limit Size Of Objects While Uploading To Amazon S3 Using Pre-Signed URL 使用预签名 URL 上传到 Amazon S3 时限制对象大小
Think about signed URL as a temporary credential for client to access a specific S3 location.将签名 URL 视为客户端访问特定 S3 位置的临时凭证。 These credential expire in a short time so there is less security concern, but do remember to restrict the access of the signed URLs appropriately
这些凭据会在短时间内过期,因此安全问题较少,但请记住适当限制签名 URL 的访问
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.