Boto3 分段上传和 md5 检查

Question

Is there a boto3 function to upload a file to S3 that verifies the MD5 checksum after upload and takes care of multipart uploads and other concurrency issues?

According to the documentation, upload_file takes care of multipart uploads and put_object can check the MD5 sum. Is there a way for me to do both without writing a long function of my own? Awscli is based on boto3 and it does that ( https://docs.aws.amazon.com/cli/latest/topic/s3-faq.html ) but I'm not sure about boto3 itself.

Answer 1

As far as I know, there is no native way in boto3 to do a multi-part upload and then easily compare md5 hashes. The answer here is to either use aws-cli or something like the code below if you want to stick with boto3 and multi-part upload (please note, this is a rough example, not production code):

import boto3
import hashlib

from botocore.exceptions import ClientError
from botocore.client import Config
from boto3.s3.transfer import TransferConfig


chunk_size=8 * 1024 * 1024

# This function is a re-worked function taken from here: https://stackoverflow.com/questions/43794838/multipart-upload-to-s3-with-hash-verification 
# Credits to user: https://stackoverflow.com/users/518169/hyperknot
def calculate_s3_etag(file_path, chunk_size=chunk_size):
    chunk_md5s = []

    with open(file_path, 'rb') as fp:
        while True:
            data = fp.read(chunk_size)

            if not data:
                break
            
            chunk_md5s.append(hashlib.md5(data))
    
    num_hashes = len(chunk_md5s)

    if not num_hashes:
        # do whatever you want to do here
        raise ValueError

    if num_hashes == 1:
        return f"{chunk_md5s[0].hexdigest()}"

    digest_byte_string = b''.join(m.digest() for m in chunk_md5s)
    digests_md5 = hashlib.md5(digest_byte_string)

    return f"{digests_md5.hexdigest()}-{num_hashes}"


def s3_md5sum(bucket_name, resource_name, client):
    try:
        return client.head_object(
            Bucket=bucket_name,
            Key=resource_name
        )['ETag'][1:-1]
    except ClientError:
        # do whatever you want to do here
        raise ClientError


bucket = "<INSERT_BUCKET_NAME>"
file = "<INSERT_FILE_NAME>"

aws_region = "<INSERT_REGION>"
aws_credentials = {
    "aws_access_key_id": "<INSERT_ACCESS_KEY>",
    "aws_secret_access_key": "<INSERT_SECRET_KEY>",
}

client = boto3.client(
    "s3", config=Config(region_name=aws_region), **aws_credentials
)
transfer_config = TransferConfig(multipart_chunksize=chunk_size)

client.upload_file(file, bucket, file, Config=transfer_config)

tag = calculate_s3_etag(file)
result = s3_md5sum(bucket, file, client)

assert tag == result

Explanation:

• During multi-part upload, the file will be split into a certain number of chunks, a hash will be calculated for each of them, combined into a byte string, and a hash of this byte string will be listed in the S3 object E-Tag as smth looking like "<hash_string>-<num_chunks>".
• What you would like to do is to essentially recreate the E-Tag locally and - after the upload - compare it with what we have in S3.
• To recreate it locally, we need to split the file in the same number of chunks (as used during the upload), calculate their hashsums, add them to a byte string, get a hash on that and then produce a string in the format of "<hash_string>-<num_chunks>".