Rails 6 Amazon S3 Error POST 403 (Forbidden) using s3_direct_upload gem

Question

I have re-written 2 applications that uses older versions of Rails v3.2 and 4.2 to current Rails 6.1.4.1 In both applications cases I have everything working properly EXCEPT for the picture upload. At first I couldn't get the s3_direct_upload to work properly. Now it appears to be working, I get the progress bars starting but I get this darn POST 403 (Forbidden) error (it has the URL in the middle of the error). The s3_direct_upload gem basically just packages up the jQuery-file-upload gem to make it easier to implement after looking at its source code.

https://github.com/waynehoover/s3_direct_upload

What is strange in my case is I know it isn't a CORS issues because I am using the very same buckets, keys and everything that are CURRENTLY STILL WORKING in the original versions of these applications that still work on Heroku. Code is virtually the same the only change is the new version of Rails. BOTH give me the exact same error. IN both apps I can view pictures just fine. IT can access the buckets with the same keys ID and everything else.

I then tried and made a sample app that uses the jQuery-file-upload gem (separate from the s3_direct_upload gem) as I wanted to verify if it was the s3_direct_upload gem causing the issue OR if it is the jQuery-file-upload gem that is causing the issues.

https://github.com/railscasts/383-uploading-to-amazon-s3/tree/master/gallery-jquery-fileupload

I made a Rails 6 version of this app and got it all working and to my horror, it is giving me the EXACT SAME ISSUE, POST 403 (Forbidden) error.
I figured out how to enable the colsole.log in the JavaScript portion of the app and got it dump the error.

Here is the full error from the console. (I put MYBUCKET in caps where my bucket name would be)

VM3692:1 POST https://MYBUCKET.s3.amazonaws.com/ 403 (Forbidden)
(anonymous) @ VM3692:1
XMLHttpRequest.send @ includes.js?v=35a79b300ab5afa978cb59af0b05e059:839
send @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:10254
ajax @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:9738
send @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13646
_onSend @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13708
(anonymous) @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:12078
data.submit @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13414
add @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:14375
_trigger @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:12478
(anonymous) @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13777
each @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:370
_onAdd @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13770
(anonymous) @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:12078
(anonymous) @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13975
fire @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:3232
add @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:3291
always @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:3400
_onChange @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:13965
(anonymous) @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:12078
handlerProxy @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:12389
dispatch @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:5226
elemData.handle @ application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:4878
application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:14438 Upload failed:
application-c24f8496b74e37ed951b517eb53daac49c7c8be5d2307e5af5413626d42a478c.js:14439 
abort: ƒ ()
add: ƒ (e, data)
autoUpload: true
bitrate: 1645390.4761904762
bitrateInterval: 500
blob: null
cache: false
contentType: false
context: jQuery.fn.init [div#upload_eq38hayegan.upload]
create: null
data: FormData {}
disabled: false
done: ƒ (e, data)
dropZone: jQuery.fn.init [div#before-pictures-dropzone.well.dropzone, context: document, selector: '#before-pictures-dropzone']
errorThrown: "Forbidden"
fail: ƒ (e, data)
fileInput: jQuery.fn.init [input#before_photo, context: input#before_photo]
fileInputClone: jQuery.fn.init [input#before_photo, prevObject: jQuery.fn.init(1), context: input#before_photo]
files: [File]
forceIframeTransport: false
form: jQuery.fn.init [form#attachment_before, context: form#attachment_before]
formAcceptCharset: "UTF-8"
formData: ƒ (form)
headers: {}
i18n: ƒ (message, context)
jqXHR: {readyState: 4, getResponseHeader: ƒ, getAllResponseHeaders: ƒ, setRequestHeader: ƒ, overrideMimeType: ƒ, …}
limitMultiFileUploadSizeOverhead: 512
loaded: 86383
messages: {uploadedBytes: 'Uploaded bytes exceed file size'}
multipart: true
originalFiles: [File]
paramName: ['file']
pasteZone: jQuery.fn.init {}
process: ƒ (resolveFunc, rejectFunc)
processData: false
processing: ƒ ()
progress: ƒ ()
progressInterval: 100
recalculateProgress: true
replaceFileInput: true
response: ƒ ()
sequentialUploads: false
singleFileUploads: true
start: ƒ (e)
state: ƒ ()
submit: ƒ ()
textStatus: "error"
timeout: 0
total: 86383
type: "POST"
uploadedBytes: 0
url: "https://MYBUCKET.s3.amazonaws.com/"
xhr: ƒ ()
_bitrateTimer: {_super: undefined, _superApply: undefined, timestamp: 1634768499443, loaded: 86383, bitrate: 1645390.4761904762, …}
_progress: {loaded: 86383, total: 86383, bitrate: 1645390.4761904762}
_response: {jqXHR: {…}, textStatus: 'error', errorThrown: 'Forbidden'}
_time: 1634768499443
[[Prototype]]: Object

I would love to know how to get these apps working right in Rails 6 if possible with the existing items. If not I am even willing to hear potential replacements preferably using the existing paperclip and amazon-sdk-v1 or amazon-sdk gems. I would just need something else to replace the upload functionality but even willing to consider something that would do all of the above. That is just a MUCH more difficult road for me as I am still a beginner at this.

I have tried both the older version like the original apps were written in amazon-sdk-v1 and amazon-sdk (version 3). Had to update some items to get the v3 to work. IN both cases I can see pictures fine, I just can't upload. The problem seems to be isolated to the POST / UPLOAD functionality and either the s3_direct_upload or the jQuery-file-upload gems. I am assuming there is something else I have to do to get these apps to work with Rails 6 I am just not aware of what it is.

If anyone wants to see anything specific please let me know, I am not sure what to show without just showing everything which I know people don't appreciate.

Any help would be appreciated, Thank You, Scott

Answer 1

The problem on this turned out to be in the way s3_direct_upload was creating the upload FORM. It was adding a part to the form with $utf8 which apparently at some point was no longer allowed. IN the s3_direct_upload gem v0.1.7 in the /app/lib/form_helper.rb in the method def policy_data method this line had to be removed from the method:

["starts-with", "$utf8", ""],

Once you removed that from your local copy of that gem the POST error goes away and it successfully uploads to Amazon S3. This however doesn't help anything but your local copy and you can't obviously push that change to Heroku or anyplace else.

Upon further examination of the s3_direct_upload GIT repository I figured out if you pull the latest's version as a plugin instead of as a gem someone along the way figured out the policy error and fixed it and they added a few more features as well but due to that you also have to add an extra option to your file_field_tag as well.

Here is the way you can pull the working version of the GEM for Rails 6

gem 's3_direct_upload', github: 'waynehoover/s3_direct_upload', ref: '6f6decc75fdf89888d7f729fc89f78e90d91cece'

If you put that in your gemfile and you use bundle install it will go to GIT and pull all of the latest changes to the GEM. It ends up putting it in a slightly different place if you go looking for your local copy of the gem. It gets put under:

C:\Ruby27-x64\lib\ruby\gems\2.7.0\bundler\gems

instead of:

C:\Ruby27-x64\lib\ruby\gems\2.7.0\gems

Lastly when you pull this latest version you have to add

data: { url: s3_uploader_url }

to the end of your file_field_tag so for example mine is:

= file_field_tag(:file, id: "before_photo", multiple: true, data: { url: s3_uploader_url })

If you pull the latest version and you add the data: { url: s3_uploader_url } to the end of your file_field_tag your s3_direct_upload should start working properly.