是否可以将文件从 URL 部分上传到 S3？ 文件.aa、文件.ab、文件.ac

Question

i would like to upload a really large file 3TB from a URL to s3 directly, but want to upload it in parts, similar to if i would split the file on my machine and upload the parts to s3.

I have the following script to upload the file (whole) from url to S3, how can i modify it to upload it to (parts) file.aa, file.ab, file.ac etc..

import requests
import boto3

url = "https://example.com/file.tar"
r = requests.get(url, stream=True)

session = boto3.Session()
s3 = session.resource('s3')

bucket_name = 'bucket-name'
key = 'file.tar' # key is the name of file on your bucket

bucket = s3.Bucket(bucket_name)
bucket.upload_fileobj(r.raw, key)

Answer 1

I would recommend reading this:

https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-python-and-boto3-9d2a0ef9b085