如何将图像数据从 s3 存储桶加载到 sagemaker 笔记本？

Question

I just started to use aws sagemaker. I tried to import images from my s3 bucket to sagemaker notebook. But I can't import images to the notebook. my image location is s3://my_bucket/train how can I import the train folder from the given path to my sagemaker notebook. I've gone through some of the solution in here and the solutions are for CSV file. All the images in my S3 bucket are in .jpeg format.

Answer 1

You don't have to download images from S3 bucket to local SageMaker instance for training the model. If you are trying to pull them for data exploration/analysis you can use aws cli from your SageMaker notebook. You can use following command to download a sample image. This will copy sample.jpg to images directory in your pwd .

aws s3 cp s3://my_bucket/train/sample.jpg ./images/sample.jpg

Try looking at amazon-sagemaker-examples repo to learn how to work with image formats on SageMaker.

Answer 2

Which SageMaker sample notebook(s) are you trying to use?

If your input data is in S3 bucket, you don't have to download it to your SageMaker notebook instance. This example shows that data is uploaded to S3 bucket: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining.ipynb

For training purposes, you can configure s3_train to your input data bucket.

And here's an inference on the image input format for the built-in image classification.

Answer 3

you could use s3fs to easily access your bucket as well as an image file in it.

import s3fs

fs = s3fs.S3FileSystem()

# To List 5 files in your accessible bucket
fs.ls('s3://bucket-name/data/')[:5]

# open it directly
with fs.open(f's3://bucket-name/data/image.png') as f:
    display(Image.open(f))

Answer 4

The snippet of code below will help you load your image folder from your s3 bucket to your SageMaker instance;

import boto3 
from botocore.exceptions import ClientError # Not necessary

# Remember to enter the cirrect bucket region below
s3 = boto3.resource('s3', region_name='us-west-2') 
# Replace the place holder with your correct bucket name
bucket = s3.Bucket('my_bucket') 
for my_bucket_object in bucket.objects.all():    
    key = my_bucket_object.key    
    print(key)    
    if not os.path.exists(os.path.dirname(key)):           
        os.makedirs(os.path.dirname(key))

# The following is basically for exception handling and not necessary to include     
    try:         
        bucket.download_file(key, key)     
    except ClientError as e:         
        if e.response['Error']['Code'] == "404":             
            print("No object with this key.")        
        else:             
            raise

Alternatively, you can also try running the following script from your Sagemaker notebook cell;

!aws s3 cp s3://$my_bucket//train/images train/images/ --recursive

The solution is from this website .

Documentation on using cp .