Python Boto3 - 如何在进程开始复制到另一个存储桶之前检查s3文件是否完全写入

Question

如何在进程B开始使用boto3将文件复制到AWS S3 Bucket B之前，确保进程A在AWS S3存储桶A中完全写入大文件（5+ GB）？

Answer 1

If a new object is being created in Amazon S3, it will only appear after the upload is complete. Other processes will not be able to view it until is has finished uploading.

Objects cannot be updated in S3. Rather, they are replaced with a new object. So, if an object is in the process of being updated, it will still appear as the old object to other processes.

The best way would be to trigger Process B by Configuring Amazon S3 Event Notifications . Once the new object is uploaded, S3 can trigger a Lambda function (or send a notification) that can then perform the second step.

Answer 2

You should definitely use s3 event notification as a trigger to a lambda function that copies your file from Bucket A to Bucket B. The trigger ensures that your copying will start once the file gets uploaded completely.

Moreover, if you have further operations to perform you can use AWS step functions in which you can define the workflow of your processes , eg process B will start after 2 seconds from process A, process C and D will execute in parallel after process B ends it's execution , etc.

Answer 3

I also do uploads of up to 40GB.

Since I do multi-part uploads, I check if the file I am writing to is closed . An S3 file(object) is only closed when all operations are finished.

Another way is to use asynchronous task queue like Celery. You will get notifications when a task is done.

I now use Golang but both those methods have worked very well for me.