How to download a file from public AWS s3 with python?

Question

I am trying to download files from a public aws s3 from this website with python scripts. For example, the first the object on the link. I tried boto3 and I got a No Credentials error:

s3 = boto3.resource('s3')
bucket = s3.Bucket('oedi-data-lake')

keys = []

for obj in bucket.objects.filter(Prefix='nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/'):
    
    if obj.key.endswith('bldg0000001-up10.zip'):
        
        keys.append(obj.key)
        
print(keys)

I also found a post Download file/folder from Public AWS S3 with Python, no credentials

and I tried as the following:

import requests

headers = {'Host' : 'oedi-data-lake.s3.amazonaws.com'}
url = 'https://oedi-data-lake.s3.amazonaws.com/nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip'
r = requests.get(url) 

but got a SSLCertVerificationError

Please help. :)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Thank you, jhashimoto!

But by doing the following, I still have the NoCredentialsError

import boto3
from botocore import UNSIGNED
from botocore.config import Config

s3 = boto3.resource("s3", config=Config(signature_version=UNSIGNED))

s3_client = boto3.client('s3')
s3_client.download_file('oedi-data-lake', 'nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip', 'bldg1.zip') 

I also read can-i-use-boto3-anonymously and changed the code as below:

import boto3
from botocore import UNSIGNED
from botocore.config import Config

client = boto3.client('s3', aws_access_key_id='', aws_secret_access_key='')
client._request_signer.sign = (lambda *args, **kwargs: None)
client.download_file('oedi-data-lake', 'nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip', 'bldg01.zip') 

and got SSLCertVerificationError.

is this something that caused by the company security policy?

Sorry for the naive questions. Completely new on AWS.

thank you so much

Answer 1

To access a bucket that allows anonymous access, configure it not to use credentials.

import boto3
from botocore import UNSIGNED
from botocore.config import Config

s3 = boto3.resource("s3", config=Config(signature_version=UNSIGNED))

# output:
# ['nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip']

python - Can I use boto3 anonymously? - Stack Overflow

Yes. Your credentials are used to sign all the requests you send out, so what you have to do is configure the client to not perform the signing step at all.

Note: Unrelated to the main topic, The AWS Python SDK team does not intend to add new features to the resource interface. You can use the client interface instead.

Resources — Boto3 Docs 1.26.54 documentation

The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3's lifecycle. Customers can find access to newer service features through the client interface.

Added at 2023/01/21 12:00:

This is a sample code using the client interface.

import boto3
from botocore import UNSIGNED
from botocore.config import Config

s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED))
s3_client.download_file('oedi-data-lake', 'nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip', 'bldg1.zip') 

Answer 2

Thank you, jhashimoto!

But by doing the following, I still have the NoCredentialsError

import boto3
from botocore import UNSIGNED
from botocore.config import Config

s3 = boto3.resource("s3", config=Config(signature_version=UNSIGNED))

s3_client = boto3.client('s3')
s3_client.download_file('oedi-data-lake', 'nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip', 'bldg1.zip') 

I also read can-i-use-boto3-anonymously and changed the code as below:

import boto3
from botocore import UNSIGNED
from botocore.config import Config

client = boto3.client('s3', aws_access_key_id='', aws_secret_access_key='')
client._request_signer.sign = (lambda *args, **kwargs: None)
client.download_file('oedi-data-lake', 'nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2022/resstock_tmy3_release_1/building_energy_models/upgrade=10/bldg0000001-up10.zip', 'bldg01.zip') 

and got SSLCertVerificationError.

is this something that caused by the company security policy?

Sorry for the naive questions. Completely new on AWS.

thank you so much