Setting DEFAULT_FILE_STORAGE in settings.py breaks app

Question

Problem:

Enabling DEFAULT_FILE_STORAGE in settings.py creates TypeError: Unicode-objects must be encoded before hashing (logs below)

• DEFAULT_FILE_STORAGE = '' creates TypeError
• DEFAULT_FILE_STORAGE = 'app.aws.utils.MediaRootS3Boto3Storage' creates TypeError

What am I doing?

• I'm trying to set up S3 storage for static & media files.
• I'm specifically trying to enable photologue storage in the /media/ folder in my AWS bucket.

What's working?

• If I comment out #DEFAULT_FILE_STORAGE = 'app.aws.utils.MediaRootS3Boto3Storage' in settings.py
• I can run python manage.py collectstatic and all my static files are successfully uploaded to the /static/ folder on aws
• Static assets are working
• Media assets are working properly locally

What's breaking?

• DEFAULT_FILE_STORAGE set to any value creates this error message.

Logs:

    Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x1103e3ea0>
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/env/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 113, in inner_run
    autoreload.raise_last_exception()
  File "/env/lib/python3.6/site-packages/django/utils/autoreload.py", line 248, in raise_last_exception
    raise _exception[1]
  File "/env/lib/python3.6/site-packages/django/core/management/__init__.py", line 327, in execute
    autoreload.check_errors(django.setup)()
  File "/env/lib/python3.6/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/env/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/env/lib/python3.6/site-packages/django/apps/registry.py", line 112, in populate
    app_config.import_models()
  File "/env/lib/python3.6/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/env/lib/python3.6/site-packages/photologue/models.py", line 76, in <module>
    "Signature: 8a477f597d28d172789f06886806bc55"))
  File "/env/lib/python3.6/site-packages/django/core/files/storage.py", line 49, in save
    return self._save(name, content)
  File "/env/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 452, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/env/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 467, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/env/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/env/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/env/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/env/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/env/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/env/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/env/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/env/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/env/lib/python3.6/site-packages/botocore/client.py", line 593, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/env/lib/python3.6/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/env/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/env/lib/python3.6/site-packages/botocore/handlers.py", line 210, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/env/lib/python3.6/site-packages/botocore/handlers.py", line 188, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/env/lib/python3.6/site-packages/botocore/handlers.py", line 202, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

Questions:

• Is this an issue with the packages I've installed?
• Why would enabling this setting DEFAULT_FILE_LOCATION trigger this error.
• What other information can I provide that would help me pin point this issue?

Packages:

boto3==1.6.4
botocore==1.9.4
Django==2.0.3
django-photologue==3.8.1
django-sortedm2m==1.5.0
django-storages==1.6.5
docutils==0.14
ExifRead==2.1.2
jmespath==0.9.3
Pillow==5.0.0
psycopg2==2.7.4
python-dateutil==2.6.1
python-decouple==3.1
pytz==2018.3
s3transfer==0.1.13
six==1.11.0

Version Numbers

• Python 3.6.4
• Django 2.0.3

Resources I've Utilized

• https://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/
• https://www.codingforentrepreneurs.com/blog/s3-static-media-files-for-django/

Thanks in advance for your help!

Answer 1

OK, so I see the second article suggests using that setting for DEFAULT_FILE_STORAGE, but it looks a bit unusual to me.

Referencing http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html , you would use:

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

for your media root.

Edit: Using Prologue with S3, it only supports Python 2.7 and the old Boto library.