Boto3 not uploading zip file to S3 python

Question

I'm trying to upload a.zip file to S3 using boto3 for python but the.zip file in my directory is not uploaded correctly. The code downloads all emails of a given user, zips them in the same directory and uploads them to an S3 bucket. The problem is that the file that gets uploaded is not the one I intend to upload. Instead a file with 18kb only appears.

Here's the code:

import sys
import imaplib
import getpass
import email
import shutil
import boto3
import os

username = input("Enter user's first name: ")
surname = input("Enter user's surname: ")
email_address = username + "." + surname + "@gmail.com"
password = getpass.getpass()
directory = username + surname + '/'
def download_emails(server):
    result, data = server.uid('search', None, "ALL")    #search all email and return their uids
    if result == 'OK':
        for num in data[0].split():
            result, data = server.uid('fetch', num, '(RFC822)')    #RFC is a standard for the format of ARPA Internet text messages
            if result == 'OK':
                email_message = email.message_from_bytes(data[0][1])    #raw email text including headers
                file_name = email_message['Subject']       #use dates and file names(can be changed)
                if not os.path.exists(directory):
                    os.makedirs(directory)      #create a dir for user's emails
                try:
                    email_file = open(directory + file_name+'.eml', 'wb')   #open a file for each email and insert the data.
                    email_file.write(data[0][1])
                    email_file.close()
                except:
                    pass

#function to zip all the emails
def archive(zipname, directory):
    return shutil.make_archive(zipname, 'zip', root_dir=directory, base_dir=None)

#function to upload zipped emails to AWS bucket
def upload_to_s3(file_name):
    s3 = boto3.resource('s3',
                aws_access_key_id=accessKey,
                aws_secret_access_key=secretKey,
                aws_session_token=secretToken,
                )

    s3.Bucket('user-backups').put_object(Key=username.title() + " " +
                                surname.title() + "/" + file_name, Body=file_name)
    print("Uploaded")


def main():
    server = imaplib.IMAP4_SSL("imap.gmail.com", 993)   #connect to gmail's imap server
    server.login(email_address, password)   #enter creds
    result, data = server.select('"[Gmail]/All Mail"')  #get all emails(inbox, outbox etc)
    if result == 'OK':
        print("Downloading")
        download_emails(server)
        server.close()
    else:
        print("ERROR: Unable to open mailbox ", result)
    server.logout()
    archive(username + surname, directory)
    upload_to_s3(username + surname + ".zip")
    #os.remove(email_address + ".zip")
    #shutil.rmtree(email_address)
    print("Done")
if __name__ == "__main__":
    main()

Answer 1

You can check out this article for more information.

There are a number of ways to upload. Check out this boto3 document where I have the methods listed below:

The managed upload methods are exposed in both the client and resource interfaces of boto3:

S3.Client method to upload a file by name: S3.Client.upload_file()
S3.Client method to upload a readable file-like object: S3.Client.upload_fileobj()
S3.Bucket method to upload a file by name: S3.Bucket.upload_file()
S3.Bucket method to upload a readable file-like object: S3.Bucket.upload_fileobj()
S3.Object method to upload a file by name: S3.Object.upload_file()
S3.Object method to upload a readable file-like object: S3.Object.upload_fileobj()

I made it work using s3.client.upload_file .

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None) .
Upload a file to an S3 object.

import boto3
s3Resource = boto3.resource('s3')

try: 
    s3Resource.meta.client.upload_file('/path/to/file', 'bucketName', 'keyName')
except Exception as err:
    print(err)

Answer 2

The put_object function accepts Body which is either bytes object or a file object. You have currently just passed the plain filename (a string).

From documentation:

Body (bytes or seekable file-like object) -- Object data.

So the fix should be to pass the file object. Consult this to know how to do that.

Answer 3

Just use s3.client.upload_file .

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

def upload_to_s3(file_name):
  s3 = boto3.client('s3')
  Key = username.title() + " " + surname.title() + "/" + file_name
  try: 
     s3.meta.client.upload_file('/path/to/file', 'user-backups', Key)
  except Exception as e:
     print(e)

Answer 4

None of the above answers worked!
The following code worked for me..

 import os 
 def upload_file_zip(s3_folder,local_file_path):
   s3_client = boto3.client('s3')
   s3_path = os.path.join(s3_folder, os.path.basename(local_file_path))
   with open(local_file_path,mode='rb') as data:
      s3_client.upload_fileobj(data, BUCKET_NAME, s3_path)

Answer 5

I managed to upload a .zip file by means of the following code:

def write_to_s3(filename, bucket, key):
    s3 = boto3.resource(service_name='s3',
                        aws_access_key_id=os.environ["AWS_USER1_ACCESS_KEY"],
                        aws_secret_access_key=os.environ["AWS_USER1_SECRET_ACCESS_KEY"])
    s3.meta.client.upload_file(filename, bucket, key)

Note: I had to use boto3.resource() instead of boto3.client() as was answered above by mootmoot, as it threw an Exception() .

Exception thrown if boto3.client() is used:

AttributeError: 'ClientMeta' object has no attribute 'client'