How to read a file from AWS S3 bucket to Python?

Question

I want to read a JSON file in AWS S3 bucket into a Python list of dicts. My AWS credentials are stored in env:

os.environ['AWS_ACCESS_KEY_ID'] = "my_access_key"
os.environ['AWS_SECRET_ACCESS_KEY'] = "my_secret_access_key"
os.environ['AWS_DEFAULT_REGION'] = 'eu-central-1'

Now I try to read the file:

f = open("s3://bucketname/folder/file.json")

And I get the following:

FileNotFoundError: [Errno 2] No such file or directory: 's3://bucketname/folder/file.json'

(I can read the file via boto3 but I wonder if there is a way for direct access like the above.)

Answer 1

The basic options are:

• Use an AWS SDK (eg boto3 for Python) to call the Amazon S3 API and download the object to your local disk. Your code can then open the local file.
• Use the smart-open · PyPI Python library, which can recognise S3 URLs and will make your code perform correctly. It calls the AWS APIs on your behalf.