Boto3 / S3 / Django：在计算文件哈希时避免阻塞事件循环

Question

From a gunicorn+gevent worker, making (multipart) uploads to S3 via django-storages , how can I make sure that during the calculation of the hash of file/part contents, the event loop is not blocked?

Looking in the source for django-storages , there doesn't really seem to be any scope for passing in the md5(/another hash), and in the source for botocore there doesn't seem to be scope for yielding the event loop while the hash is being calculated.

Answer 1

There is now a PR for django-storages that calculates the MD5 hash as data received by the application .

In summary, this changes S3Boto3StorageFile so on init and new part

self._file_md5 = hashlib.md5()

and then on receive of content_bytes

self._file_md5.update(content_bytes)

and then when each part is uploaded, the ContentMD5 parameter is passed to the part upload function

ContentMD5=base64.b64encode(self._file_md5.digest()).decode('utf-8')