如何使用 Python 从 Amazon S3 下载 JPG 文件到 memory？

Question

I'm struggling to download a JPG file from Amazon S3 using Python, I want to load this code onto Heroku so I need to the image to be loaded into memory rather than onto disk.

The code I'm using is:

import boto3

s3 = boto3.client(
        "s3",
        aws_access_key_id = access_key,
        aws_secret_access_key = access_secret
    )

s3.upload_fileobj(image_conv, bucket, Key = "image_3.jpg")

new_obj = s3.get_object(Bucket=bucket, Key="image_3.jpg")

image_dl = new_obj['Body'].read()

Image.open(image_dl)

I'm getting the error message:

File ..... line 2968, in open
    fp = builtins.open(filename, "rb")
ValueError: embedded null byte

Calling image_dl returns a massive long list of what I assume are bytes, one small section looks like the following:

f\xbc\xdc\x8f\xfe\xb5q\xda}\xed\xcb\xdcD\xab\xe6o\x1c;\xb7\xa0\xf5\xf5\xae\xa6)\xbe\xee\xe6\xc3vn\xdfLVW:\x96\xa8\xa3}\xa4\xd8\xea\x8f*\x89\xd7\xcc\xe8\xf0\xca\xb9\x0b\xf4\x1f\xe7\x15\x93\x0f\x83ty$h\xa6\x83\xc8\x99z<K\xc3c\xd4w\xae\xa4\xc2\xfb\xcb\xee\xe0

The image before I uploaded to S3 returned the below and that's the format that I'm trying to return the image into. Is anyone able to help me on where I'm going wrong?

<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1440x1440 at 0x7F2BB4005EB0>

Answer 1

Pillow's Image class needs either a filename to open, or a file-like object that it can call read on. Since you don't have a filename, you'll need to provide a stream. It's easiest to use BytesIO to turn the byte array into a strem:

import boto3
from PIL import Image
from io import BytesIO

bucket = "--example-bucket--"

s3 = boto3.client("s3")
with open("image.jpg", "rb") as image_conv:
    s3.upload_fileobj(image_conv, bucket, Key="image_3.jpg")
new_obj = s3.get_object(Bucket=bucket, Key="image_3.jpg")
image_dl = new_obj['Body'].read()
image = Image.open(BytesIO(image_dl))

print(image.width, image.height)

Answer 2

Try first to load raw data into a BytesIO container:

from io import StringIO
from PIL import Image

file_stream = StringIO()
s3.download_fileobj(bucket, "image_3.jpg", file_stream)
img = Image.open(file_stream)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.download_fileobj