how to load image data from the bucket to AWS sagemaker notebook?

Question

The images are present as folders - train and test in my s3 bucket. I want to use them as it is in my sagemaker notebook. For example, like on my local server I use test_dir = "C:\\Users\\catvdog\\dataset\\test".

Answer 1

An example of a dir would be:

's3://sagemaker-eu-central-1-123456789/test/sagemaker/sklearncontainer/test'

In more general, you create your path for a file

data_file = 'filename.csv'
datafolder = 'xxx/yyy'
data_location = 's3://{}/{}/{}'.format(bucket, datafolder, data_file)

Answer 2

I think what you're looking for is the s3fs module. For example:

import s3fs

fs = s3fs.S3FileSystem()

directory = 'bucket_name/data'
filenames = fs.ls(directory)
for filename in filenames:
    with fs.open(filename, 'rb') as f:
        # do something with file

For more information s3fs documentation