Django-storages not detecting changed static files

Question

I'm using django-storages and amazon s3 for my static files. Following the documentation, I put these settings in my settings.py

STATIC_URL = 'https://mybucket.s3.amazonaws.com/'

ADMIN_MEDIA_PREFIX = 'https://mybucket.s3.amazonaws.com/admin/'

INSTALLED_APPS += (
    'storages',
)

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'mybucket_key_id'
AWS_SECRET_ACCESS_KEY = 'mybucket_access_key'
AWS_STORAGE_BUCKET_NAME = 'mybucket'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

And the first time I ran collect static everything worked correctly and my static files were uploaded to my s3 bucket.

However, after making changes to my static files and running python manage.py collectstatic this is outputted despite the fact that static files were modified

-----> Collecting static files
    0 static files copied, 81 unmodified.

However, if I rename the changed static file, the changed static file is correctly copied to my s3 bucket.

Why isn't django-storages uploading my changed static files? Is there a configuration problem or is the problem deeper?

Answer 1

collectstatic skips files if "target" file is "younger" than source file. Seems like amazon S3 storage returns wrong date for you file.

you could investigate [code][1] and debug server responses. Maybe there is a problem with timezone.

Or you could just pass --clear argument to collectstatic so that all files are deleted on S3 before collecting

Answer 2

https://github.com/antonagestam/collectfast

From readme.txt : Custom management command that compares the MD5 sum and etag from S3 and if the two are the same skips file copy. This makes running collect static MUCH faster if you are using git as a source control system which updates timestamps.

Answer 3

Create a settings file just for collectstatic sync, with this config:

TIME_ZONE = 'UTC'

Run collectstatic with a specific settings with this line:

python manage.py collectstatic --settings=settings.collectstatic

Answer 4

This question is a little old but in case it helps someone in the future, I figured I'd share my experience. Following advice found in other threads I confirmed that, for me, this was indeed caused by a difference in time zone. My django time wasn't incorrect but was set to EST and S3 was set to GMT. In testing, I reverted to django-storages 1.1.5 which did seem to get collectstatic working. Partially due to personal preference, I was unwilling to a) roll back three versions of django-storages and lose any potential bug fixes or b) alter time zones for components of my project for what essentially boils down to a convenience function (albeit an important one).

I wrote a short script to do the same job as collectstatic without the aforementioned alterations. It will need a little modifying for your app but should work for standard cases if it is placed at the app level and 'static_dirs' is replaced with the names of your project's apps. It is run via terminal with 'python whatever_you_call_it.py -e environment_name (set this to your aws bucket).

import sys, os, subprocess
import boto3
import botocore
from boto3.session import Session
import argparse
import os.path, time
from datetime import datetime, timedelta
import pytz

utc = pytz.UTC
DEV_BUCKET_NAME = 'dev-homfield-media-root'
PROD_BUCKET_NAME = 'homfield-media-root'
static_dirs = ['accounts', 'messaging', 'payments', 'search', 'sitewide']

def main():
    try: 
        parser = argparse.ArgumentParser(description='Homfield Collectstatic. Our version of collectstatic to fix django-storages bug.\n')
        parser.add_argument('-e', '--environment', type=str, required=True, help='Name of environment (dev/prod)')
        args = parser.parse_args()
        vargs = vars(args)
        if vargs['environment'] == 'dev':
            selected_bucket = DEV_BUCKET_NAME
            print "\nAre you sure? You're about to push to the DEV bucket. (Y/n)"
        elif vargs['environment'] == 'prod':
            selected_bucket = PROD_BUCKET_NAME
            print "Are you sure? You're about to push to the PROD bucket. (Y/n)"
        else:
            raise ValueError

        acceptable = ['Y', 'y', 'N', 'n']
        confirmation = raw_input().strip()
        while confirmation not in acceptable:
            print "That's an invalid response. (Y/n)"
            confirmation = raw_input().strip()

        if confirmation == 'Y' or confirmation == 'y':
            run(selected_bucket)
        else:
            print "Collectstatic aborted."
    except Exception as e:
        print type(e)
        print "An error occured. S3 staticfiles may not have been updated."


def run(bucket_name):

    #open session with S3
    session = Session(aws_access_key_id='{aws_access_key_id}',
        aws_secret_access_key='{aws_secret_access_key}',
        region_name='us-east-1')
    s3 = session.resource('s3')
    bucket = s3.Bucket(bucket_name)

    # loop through static directories
    for directory in static_dirs:
        rootDir = './' + directory + "/static"
        print('Checking directory: %s' % rootDir)

        #loop through subdirectories
        for dirName, subdirList, fileList in os.walk(rootDir):
            #loop through all files in subdirectory
            for fname in fileList:
                try:
                    if fname == '.DS_Store':
                        continue

                    # find and qualify file last modified time
                    full_path = dirName + "/" + fname
                    last_mod_string = time.ctime(os.path.getmtime(full_path))
                    file_last_mod = datetime.strptime(last_mod_string, "%a %b %d %H:%M:%S %Y") + timedelta(hours=5)
                    file_last_mod = utc.localize(file_last_mod)

                    # truncate path for S3 loop and find object, delete and update if it has been updates
                    s3_path = full_path[full_path.find('static'):]
                    found = False
                    for key in bucket.objects.all():
                        if key.key == s3_path:
                            found = True 
                            last_mode_date = key.last_modified
                            if last_mode_date < file_last_mod:
                                key.delete()
                                s3.Object(bucket_name, s3_path).put(Body=open(full_path, 'r'), ContentType=get_mime_type(full_path))
                                print "\tUpdated : " + full_path
                    if not found:
                        # if file not found in S3 it is new, send it up
                        print "\tFound a new file. Uploading : " + full_path
                        s3.Object(bucket_name, s3_path).put(Body=open(full_path, 'r'), ContentType=get_mime_type(full_path))
                except:
                    print "ALERT: Big time problems with: " + full_path + ". I'm bowin' out dawg, this shitz on u." 


def get_mime_type(full_path):
    try:
        last_index = full_path.rfind('.')
        if last_index < 0:
            return 'application/octet-stream'
        extension = full_path[last_index:]
        return {
            '.js' : 'application/javascript',
            '.css' : 'text/css',
            '.txt' : 'text/plain',
            '.png' : 'image/png',
            '.jpg' : 'image/jpeg',
            '.jpeg' : 'image/jpeg',
            '.eot' : 'application/vnd.ms-fontobject',
            '.svg' : 'image/svg+xml',
            '.ttf' : 'application/octet-stream',
            '.woff' : 'application/x-font-woff',
            '.woff2' : 'application/octet-stream'
        }[extension]
    except:
        'ALERT: Couldn\'t match mime type for '+ full_path + '. Sending to S3 as application/octet-stream.'

if __name__ == '__main__':
    main()