Creating Django objects with objects in an S3 bucket

Question

I've uploaded a large amount of images to S3, using the awcli utility. I am trying to write a Django management command to iterate over them and create objects in the database for them.

So far, I've been able to get the object - but I'm stuck on what I should do to make the appropriate Django object.

Obviously this code doesn't work but hopefully sheds some light on what I am trying to do.

s3 = boto3.resource('s3', region_name='us-east-2')
bucket = s3.Bucket('photo-uploads')
object = bucket.Object('00004542/000045420020.jpg')

photo = Photo.objects.create(title='Some title', image=object)
photo.save()

Answer 1

I did this with a few extra steps, but i also made the assumption that since we have the objects stored in s3 we don't want to also store them as a BLOB in the database. I'm only storing all the references we need to access the object on a given resource:

here are your models:

class Resource(models.Model):
    ...
    region_name = models.CharField(max_length=32, default='us-east-2')
    bucket_name = models.CharField(max_length=32, default='photo-uploads')

class Photo(models.Model):
    ...
    title = models.CharField(max_length=56)
    object_key = models.CharField(max_length=128) # make this however big you need to fit your worst case object key string
    resource = models.ForeignKey(Resource, on_delete=models.DO_NOTHING)

The ellipses are where you would put your primary key soup de jour (UUID, AutoField, etc)

With this we can ensure that we have all the information needed to retrieve the image from the bucket or generate a URL for the object with the object key and the bucket information.

You dont need to store the physical s3 object in your database since you're already doing that in s3.

Let me know if this makes sense.

Then you would do this

resource = Resource.objects.create(region_name='us-east-2', bucket_name='photo-uploads')
photo = Photo.objects.create(title="Some title", object_key='00004542/000045420020.jpg', resource=resource)

Which then yields us the ability to download the file from the info stored in the database using the Photo model.

import boto3
from .models import Photo

# get the first one since in my example we only have one, but you would use the Photo id for other usecases
photo = Photo.objects.first()

s3 = boto3.resource('s3', region_name=photo.resource.region_name)
bucket = s3.Bucket(photo.resource.bucket_name)
object = bucket.Object(photo.object_key)