使用 boto3 将文件上传到 S3

Question

I have a script to upload a csv file which is in a container to S3 bucket, I copied the file to my local machine and I'm testing the script locally, but getting errors. I'm still learning everything, trying to know what part I'm missing in the script and how I can get this running and upload the file to S3,

Here's the errors:

error_1:

Traceback (most recent call last):
  File "C:/Users/U12345/IdeaProjects/xxx/s3_upload.py", line 19, in <module>
    r'C:\Users\U12345\IdeaProjects\xxx\test_' + str(current_date) + '.csv')
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\\Users\\U12345\\IdeaProjects\\xxx\\test.csv' -> 'C:\\Users\\U12345\\IdeaProjects\\xxx\\test_2020-04-16 10:55:41.csv'

error_02:

File "C:/Users/U12345/IdeaProjects/xxx/s3_upload.py", line 33
    response = s3_client.put_object(Body='C:\Users\U12345/IdeaProjects/xxx/test.csv',
                                        ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

Another issue is I'm not very sure how to call this function, what parameter to put in the bracket, it gave me different errors.

I've been struggling with this for almost a week now, a bit frustrated, can someone gave me some help or a good example that I can follow.

Update:

error02 and the last issue have been solved, it's just the first error still not working, I've trying '/', '', with 'C:', without 'C:', all not working...

Answer 1

I guess you are using put_object() the wrong way. It is used to save an 'object' on s3 & not a file ie you need to first read the file content using pandas.read_csv() or something else & then replace the 'Body' part with the object obtained on reading.Something like this

  df= pandas.read_csv('C:\Users\U12345/IdeaProjects/xxx/test.csv')
  response = s3_client.put_object(Body=df,  
                                    Bucket=output_bucket,
                                    Key='test.csv',
                                    ACL="bucket-owner-full-control")

If you wish to upload the file directly, you should use

  s3 = boto3.resource('s3')
  s3.meta.client.upload_file('C:\Users\U12345/IdeaProjects/xxx/test.csv', output_bucket, 'test.csv')

Answer 2

You've got a few things to address here so lets break it down a little bit.

1) When you call upload_to_s3() you need to call it with the function parameters you've declared it with, a filename and a bucket key. So it would be upload_to_s3(filename, bucket_key) for example.

2) It's a been a while since I used Windows & Python but ask yourself if it uses \ instead of / in file paths, also make sure the file is definitely in the location you expect.

3) For the S3 upload the Body: is the actual data you want to upload, not the filename of the data. You have called it inside open(...) as file so you now have an object called file that represents it.

Answer 3

in the last line

upload_to_s3()

you haven't actually given the function any parameters. inside the brackets put in the params:

(source_filename: str, key: str)

ie give the function it's filename and bucket