将媒体文件和静态文件从 Amazon S3 提供给 Django/Heroku

Question

I've got a question about serving media files and static files from an S3 bucket. I'd be extremely grateful if anyone could give some pointers about how to solve the problem.

Background

• I've completed a number of tutorials to get a basic idea for how Django works. To cement what I've learnt, I've decided to repurpose a lot of the concepts into a simple web app and deploy it online.

https://tbdcl-allconf-production.herokuapp.com/

https://www.allconf.io/

• The site shows a list of conferences , which have a one-to-many relationship with a conference series (eg DjangoCon parent with DjangoCon 2019 and DjangoCon 2020 as children ).
• The conference series has a logo image as part of its model. The image is displayed as part of a card in a list of all conferences:

https://tbdcl-allconf-production.herokuapp.com/conferences/

https://www.allconf.io/conferences/

• These are all added through the admin, as it's not intended for users to be able to update the conference list.

Problem

• Everything worked fine when working locally, BUT when I deployed to Heroku the logos would not display (even though the files were visible in the admin view in the deployed app).
• A quick search revealed that Heroku cannot serve media files, so I chose the (seemingly) most common approach – adding boto3 and django-storages to my project, and pointing it at an Amazon S3 bucket.
• I followed a number of popular tutorials to set this up.

• However it looks like the app is still looking at "itself" for the media and static files, not the S3 bucket. I can tell this because:

• Simply, the media files and static files are not in the S3 bucket, even though I've followed the tutorials and added the AWS credentials to Django.

• When I open the developer view in Chrome, the source for the images + CSS files looks like the local path (there's no reference to "aws" or "s3" in each file's path).

Possible solutions

• After doing some more digging + basic troubleshooting, I think it could be a couple of things:
  1. Whitenoise is still set up in my project (from when I was working locally with only static files). Should this be removed? Is it conflicting with boto3 and django-storages?
  2. Do I need to update my models in some way (eg with the def __save__ method)?
  3. Do I need to migrate the database (or any other database task) to update the URLs for each model's image file?
  4. Is the fact that the app is running in a Docker container causing an issue with communicating "out" to the S3 bucket?

Although to be honest, I'm completely lost! I can't find any resources online that explain this issue and how to resolve it.

I'd be extremely grateful for any advice.

If you need to see the settings.py and conferences/models.py files, I've pasted them below.

Many thanks!

settings.py

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

ENVIRONMENT = os.environ.get('ENVIRONMENT', default='production')

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.environ.get('SECRET_KEY')

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = int(os.environ.get('DEBUG', default=0))

ALLOWED_HOSTS = ['tbdcl-allconf-production.herokuapp.com', '.allconf.io', 'localhost', '127.0.0.1']


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'whitenoise.runserver_nostatic',
    'django.contrib.staticfiles',

    # Third party
    'debug_toolbar',
    'storages',

    # Local
    'users.apps.UsersConfig',
    'pages.apps.PagesConfig',
    'conferences.apps.ConferencesConfig',
]

MIDDLEWARE = [
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
]

CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 600
CACHE_MIDDLEWARE_KEY_PREFIX = ''

ROOT_URLCONF = 'allconf_project.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'allconf_project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'SQL_NAME',
        'USER': 'SQL_USER',
        'PASSWORD': 'SQL_PASSWORD',
        'HOST': 'db',
        'PORT': 5432
    }
}

# Password validation
# https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/2.2/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'America/New_York'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.2/howto/static-files/

USE_S3 = os.environ.get('USE_S3') == 'TRUE'

if USE_S3:
    # aws settings
    AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
    AWS_S3_FILE_OVERWRITE = False
    AWS_DEFAULT_ACL = None
    AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
    # s3 static settings
    STATIC_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'hello_django.storage_backends.StaticStorage'
    # s3 public media settings
    PUBLIC_MEDIA_LOCATION = 'media'
    MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'hello_django.storage_backends.PublicMediaStorage'
    # s3 private media settings
    PRIVATE_MEDIA_LOCATION = 'private'
    PRIVATE_FILE_STORAGE = 'hello_django.storage_backends.PrivateMediaStorage'
else:
    STATIC_URL = '/staticfiles/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    MEDIA_URL = '/mediafiles/'
    MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')

STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)
STATICFILES_FINDERS = [
    "django.contrib.staticfiles.finders.FileSystemFinder",
    "django.contrib.staticfiles.finders.AppDirectoriesFinder",
]


# User authentication

AUTH_USER_MODEL = 'users.CustomUser'


# django-debug-toolbar

import socket
hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
INTERNAL_IPS = [ip[:-1] + "1" for ip in ips]


# production

if ENVIRONMENT == 'production':
    SECURE_BROWSER_XSS_FILTER = True
    X_FRAME_OPTIONS = 'DENY'
    SECURE_SSL_REDIRECT = True
    SECURE_HSTS_SECONDS = 3600
    SECURE_HSTS_INCLUDE_SUBDOMAINS = True
    SECURE_HSTS_PRELOAD = True
    SECURE_CONTENT_TYPE_NOSNIFF = True
    SESSION_COOKIE_SECURE = True
    CSRF_COOKIE_SECURE = True
    SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')


# heroku

import dj_database_url
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)

storage_backends.py

from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings


class StaticStorage(S3Boto3Storage):
    location = 'static'
    default_acl = 'public-read'


class PublicMediaStorage(S3Boto3Storage):
    location = 'media'
    default_acl = 'public-read'
    file_overwrite = False


class PrivateMediaStorage(S3Boto3Storage):
    location = 'private'
    default_acl = 'private'
    file_overwrite = False
    custom_domain = False

conferences/models.py

import uuid
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import models
from django.urls import reverse


class Series(models.Model):
    slug = models.SlugField(null=False, unique=True)
    title = models.CharField(max_length=200)
    description = models.CharField(max_length=300)
    series_logo = models.ImageField(upload_to='series_logos/', blank=True)

    class Meta: 
        indexes = [
            models.Index(fields=['slug'], name='slug_index'),
        ]

    def __str__(self):
        return self.title

    def get_absolute_url(self):
        return reverse('series_detail', kwargs={'slug': self.slug})


class Conference(models.Model):
    id = models.UUIDField(
        primary_key=True,
        default=uuid.uuid4,
        editable=False,
    )
    title = models.CharField(max_length=255)
    start_date = models.DateField()
    end_date = models.DateField()
    location = models.CharField(max_length=255)
    conf_url = models.CharField(max_length=255)
    conf_series = models.ForeignKey(
        Series,
        null=True,
        on_delete=models.CASCADE,
        related_name='conferences',
    )

    def __str__(self):
        return self.title

    def get_absolute_url(self):
        return reverse('conference_detail', args=[str(self.id)])

Answer 1

Well it looks like you still have whitenoise in middleware, so try removing that. I would also double AWS_S3_CUSTOM_DOMAIN against what is shown in AWS. I believe the region should be included in there as well.

For me, I had this issue seemingly because of django-heroku. I had to update my settings.py file with the following:

django_heroku.settings(locals(), staticfiles=False)