Unable to view files on aws s3 (403 Forbidden)

Question

I am able to upload to AWS S3 using django-storages & boto3 but can't view (403 Forbidden).

I followed the guide on: https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html

And I have successfully set up buckets and transferred my static files to Amazon S3.

However when I try to load the S3 files, I get a 403 Forbidden message.

Here is what is in my settings.py file:

# AWS s3 (image storage)

AWS_ACCESS_KEY_ID = 'XXX' # XXX replaced with the actual key
AWS_SECRET_ACCESS_KEY = 'YYY' # YYY replaced with the actual key

# sensitive data will be replaced with environment variables when page is public

AWS_STORAGE_BUCKET_NAME = 'diceart-static'
AWS_S3_CUSTOM_DOMAIN = 's3.eu-west-2.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
AWS_DEFAULT_ACL = None
AWS_BUCKET_ACL = None

AWS_QUERYSTRING_AUTH = False

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'static'),
]

STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

So when I ran

python manage.py collectstatic

Everything worked! And I could see all of my files uploaded to Amazon S3. I can also see the time of the programmatic access of this so I know it all worked.

Here is the contents of my html template file where I'm hoping to serve the S3 files:

{% extends 'dice/base.html' %}
{% load crispy_forms_tags %}
{% load static %}

{% block content %}
  <form method="post" enctype="multipart/form-data">
    {% csrf_token %}
    {{ form|crispy }}
    {% if not uploaded_file_url %}
        <button type="submit" class="btn btn-success mt-2">Upload</button>
    {% endif %}
  </form>

  {% if uploaded_file_url %}
    <p>File uploaded at: <a href="{{ uploaded_file_url }}">{{ uploaded_file_url }}</a></p>
    <img src="{{ uploaded_file_url }}">
  {% endif %}

  <p><a href="{% url 'index' %}">Return to home</a></p>
  <p>Have a picture of a cat:</p>
  <p><img src="{% static 'cat.jpg' %}"></p>
{% endblock %}

The key bit here is the penultimate line:

<p><img src="{% static 'cat.jpg' %}"></p>

The code above relates to user file uploads, which I am not doing using S3 for the time being.

Now when the page loads it does so without error and the relevant line shows like this:

<img src="https://s3.eu-west-2.amazonaws.com/diceart-static/static/cat.jpg">

And checking this against my S3 bucket it appears to match:

https://s3.eu-west-2.amazonaws.com/diceart-static/static/cat.jpg

But I get a 403 Forbidden error and no image loads.

Here is what I can see in the Chrome developer tools network headers:

Request URL: https://s3.eu-west-2.amazonaws.com/diceart-static/static/cat.jpg
Request Method: GET
Status Code: 403 Forbidden
Remote Address: 52.95.148.64:443
Referrer Policy: no-referrer-when-downgrade
Provisional headers are shown
Referer: https://mysubdomain.pythonanywhere.com/upload2/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

The referrer is the page which loads the image, hosted on Pythonanywhere.

Now something that I am unsure of is the url of my S3 bucket. Everywhere I check online seems to have the structure

https://diceart-static.s3.amazonaws.com

rather than what I have:

https://s3.eu-west-2.amazonaws.com/diceart-static

I think this may just be a change to AWS's url structure and not the issue here as my uploads worked, but it is worth mentioning perhaps.

So essentially I can upload programatically but I can't view my S3 files using django-storages and boto3. What am I missing here?

Note that there is no public access allowed for the bucket, but I can't change this because Amazon doesn't allow me to(?!). I didn't think this would matter as I'm putting in my credentials here which I thought would be sent as headers to give me access.

Answer 1

From the above setting AWS_DEFAULT_ACL is set to None and the doc says

If set to None then all files will inherit the bucket's ACL.

and the default ACL of S3 is private .

You can set AWS_DEFAULT_ACL = None as recommended and then write your own storage class by inheriting S3Boto3Storage :

#myapp/myfilestorage.py

from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings

class StaticS3Storage(S3Boto3Storage):
   location = settings.AWS_STATIC_LOCATION
   default_acl = 'public-read'

Then you we also have this in your settings:

AWS_STATIC_LOCATION = 'static'

STATICFILES_STORAGE = 'myapp.myfilestorage.StaticS3Storage'

you can also edit/duplicate it for DEFAULT_FILE_STORAGE as the case may be

Answer 2

I think that article is dated so is a bit misleading. django-storages docs aren't great, but from some trial and error, I think the settings you actually want are:

AWS_ACCESS_KEY_ID = '<enter key>'
AWS_SECRET_ACCESS_KEY = '<enter key>'
AWS_STORAGE_BUCKET_NAME = 'diceart-static'
AWS_S3_REGION_NAME = 'eu-west-2'
AWS_DEFAULT_ACL = None
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'static'),
]

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

Note that I've removed both AWS_S3_CUSTOM_DOMAIN and STATIC_URL from the settings. Those will be built automatically for you by django-storages .

With these settings, you should still be able to leave your bucket permissions as not available to public and it should still work because it uses querystring auth.

---

Edit

While the above should work, I'm not sure it's a "best practice" for a public web app. If you are able to uncheck the Block all public access box and turn that off, you can use these settings instead, and querystring auth won't be used (or needed).

# ... same as above
# Make default ACL publicly readable
AWS_DEFAULT_ACL = 'public-read'
# Turn off query auth, although not sure this is needed.
AWS_QUERYSTRING_AUTH = False
# ... same as above