Get a specific file from s3 bucket (boto3)

Question

So I have a file.csv on my bucket 'test', I'm creating a new session and I wanna download the contents of this file:

session = boto3.Session(
aws_access_key_id=KEY,
aws_secret_access_key=SECRET_KEY
)
s3 = session.resource('s3')
obj = s3.Bucket('test').objects.filter(Prefix='file.csv')

This returns me a collection but is there a way to fetch the file directly? Without any loops, I wanna do something like:

s3.Bucket('test').objects.get(key='file.csv')

I could achieve the same result without passing credentials like this:

s3 = boto3.client('s3')
obj = s3.get_object(Bucket='test', Key='file.csv')

Answer 1

If you take a look at the client method:

import boto3

s3_client = boto3.client('s3')
s3_client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

and the resource method:

import boto3

s3 = boto3.resource('s3')
s3.meta.client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

you'll notice that you can convert from the resource to the client with meta.client .

So, combine it with your code to get:

session = boto3.Session(aws_access_key_id=KEY, aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
obj = s3.meta.client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

Answer 2

I like mpu.aws.s3_download , but I'm biased ;-)

It does it like that:

import os
import boto3

def s3_download(bucket_name, key, profile_name, exists_strategy='raise'):
    session = boto3.Session(profile_name=profile_name)
    s3 = session.resource('s3')
    if os.path.isfile(destination):
        if exists_strategy == 'raise':
            raise RuntimeError('File \'{}\' already exists.'
                               .format(destination))
        elif exists_strategy == 'abort':
            return
    s3.Bucket(bucket_name).download_file(key, destination)

For authentication, I recommend using environment variables. See boto3: Configuring Credentials for details.

Answer 3

you can use the following boto3 method.

download_file(Bucket, Key, Filename, ExtraArgs=None, Callback=None, Config=None)

s3 = boto3.resource('s3')
s3.meta.client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

find more details here - download_file()