Error 500 while Uploading CSV file to S3 bucket using boto3 and python flask

Question

kind of looked at all possible options. I am using boto3 and python3.6 to upload file to s3 bucket, Funny thing is while json and even .py file is getting uploaded, it is throwing Error 500 while uploading CSV. On successful uplaod i am returning an json to check all the values.

    import boto3
    from botocore.client import Config

    @app.route("/upload",methods = ['POST','GET'])
    def upload(): 
    if request.method == 'POST':
        file = request.files['file']
        filename = secure_filename(file.filename)
        s3 = boto3.resource('s3', aws_access_key_id= os.environ.get('AWS_ACCESS_KEY_ID'), aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY'),config=Config(signature_version='s3v4'))
        s3.Bucket(os.environ.get('S3_BUCKET')).put_object(Key=filename, Body=open(filename, 'rb'), ContentEncoding='text/csv')
        return jsonify({'successful upload':filename, 'S3_BUCKET':os.environ.get('S3_BUCKET'), 'ke':os.environ.get('AWS_ACCESS_KEY_ID'), 'sec':os.environ.get('AWS_SECRET_ACCESS_KEY'),'filepath': "https://s3.us-east-2.amazonaws.com/"+os.environ.get('S3_BUCKET')+"/" +filename})

Please help!!

Answer 1

You are getting a FileNotFoundError for file xyz.csv because the file does not exist.

This could be because the code in upload() does not actually save the uploaded file, it merely obtains a safe name for it and immediately tries to open it - which fails.

That it works for other files is probably due to the fact that those files already exist, perhaps left over from testing, so there is no problem.

Try saving the file to the file system using save() after obtaining the safe filename:

upload_file = request.files['file']
filename = secure_filename(upload_file.filename)
upload_file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))

and then uploading it (assuming that you've configured an UPLOAD_FOLDER ):

with open(os.path.join(app.config['UPLOAD_FOLDER'], filename), 'rb') as f:
    s3.Bucket(os.environ.get('S3_BUCKET')).put_object(Key=filename, Body=f, ContentEncoding='text/csv')
    return jsonify({...})

---

There is no need to actually save the file to the file system; it can be streamed directly to your S3 bucket using the stream attribute of the upload_file object:

upload_file = request.files['file']
filename = secure_filename(upload_file.filename)

s3 = boto3.resource('s3', aws_access_key_id='key', aws_secret_access_key='secret')
s3.Bucket('bucket').put_object(Key=filename, Body=upload_file.stream, ContentType=upload_file.content_type)

To make this more generic you should use the content_type attribute of the uploaded file as shown above.