AWS S3 file uploads returning 403:Forbidden with Carrierwave/fog gems in Rails

Question

I've been scratching my head over this for several days. Currently working through Chapter 13 (user microposts) of the Rails Tutorial , and while my app works fine in development, I cannot seem to get the image uploads to AWS S3 running in production (details here in the tutorial ). The app utililzes ASW S3 buckets for storage, and CarrierWave /Fog gems for file upload.

When I open my production heroku app, everything works fine except the image upload. When I try to upload an image on a new micropost, I get a generic 'We're sorry, but something went wrong.' error in the browser. Heroku logs show a :status 403 error "Forbidden" when trying to reach the bucket (detailed logs below).

Others have seemed to have similar problems. In most cases it seems the solution is to set proper IAM user permissions or set the S3 bucket policy , but I am fairly confident I have those set up correctly for two reasons:

1. I am able to use the web GUI on that user account to upload and manage files in the bucket, so I know it has access.
2. I installed the aws command line tool, configured it with the same user access key and secret key used in my web app, and I am able to upload and retrieve information from the bucket through the CLI.

Still I have tried several bucket policies and user permissions, including the Amazon S3 full access permission (I believe this is the most general), and several more specific versions (see latest below).

Other things I have tried that seem to work for others:

• Updating CORS configuration for the bucket (although honestly I do not fully understand CORS or if it is necessary for accessing buckets. Again I started with a general config, not a specific one, and still can't get it to work).
• Using different gems for storage , such as carrierwave-aws in lieu of fog.
• Setting fog's config.delete_tmp_file_after_storage = false -- two examples of success for this: 1 | 2

Being a new developer, I feel I don't have the technical sophistication yet to diagnose this one, and frankly it is a bit discouraging, so I am hoping for help. Here are some questions I have and possible lines of inquiry I would like to look further into:

• Question: If I am able to access the bucket with the aws-cli, shouldn't I also be able to use heroku with carrierwave/fog using the same credentials, or am I misunderstanding something about how heroku reaches for the S3 bucket?
• Possible issue: Carrierwave or ImageMagick creates a temp file when the image is uploaded. In some cases it seems that may interfere with the bucket upload. Someone gave a vague answer about this here , but I do not understand what action might help get rid of that problem.
• Possible issue: Someone seemed to have issues with Rails strong params while doing this. I could not quite understand how to debug this... When I use byebug to open an interactive console in the server at the error location (in my the micropost_controller #create method), right after the file upload, the image is in the params hash, but the :picture key of the instance variable @micropost is nil (at the same time, the :content key is not nil, it contains whatever text I submitted with the picture, as it should).

Link to the rest of the code on GitHub if it helps.

Sorry for the long-windedness. Any guidance would be greatly appreciated.

---

Errors:

2019-01-07T08:11:37.684069+00:00 app[web.1]: F, [2019-01-07T08:11:37.683926 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737] Excon::Error::Forbidden (Expected(200) <=> Actual(403 Forbidden)
2019-01-07T08:11:37.684074+00:00 app[web.1]: excon.error.response
2019-01-07T08:11:37.684077+00:00 app[web.1]: :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>***</RequestId><HostId>***=</HostId></Error>"
2019-01-07T08:11:37.684079+00:00 app[web.1]: :cookies       => [
2019-01-07T08:11:37.684081+00:00 app[web.1]: ]
2019-01-07T08:11:37.684082+00:00 app[web.1]: :headers       => {
2019-01-07T08:11:37.684085+00:00 app[web.1]: "Connection"       => "close"
2019-01-07T08:11:37.684086+00:00 app[web.1]: "Content-Type"     => "application/xml"
2019-01-07T08:11:37.684088+00:00 app[web.1]: "Date"             => "Mon, 07 Jan 2019 08:11:36 GMT"
2019-01-07T08:11:37.684090+00:00 app[web.1]: "Server"           => "AmazonS3"
2019-01-07T08:11:37.684092+00:00 app[web.1]: "x-amz-id-2"       => "***"
2019-01-07T08:11:37.684094+00:00 app[web.1]: "x-amz-request-id" => "***"
2019-01-07T08:11:37.684096+00:00 app[web.1]: }
2019-01-07T08:11:37.684098+00:00 app[web.1]: :host          => "bucket-name.s3-us-west-1.amazonaws.com"
2019-01-07T08:11:37.684099+00:00 app[web.1]: :local_address => "*********"
2019-01-07T08:11:37.684101+00:00 app[web.1]: :local_port    => ******
2019-01-07T08:11:37.684103+00:00 app[web.1]: :path          => "/uploads/micropost/picture/306/ocean2.jpeg"
2019-01-07T08:11:37.684104+00:00 app[web.1]: :port          => 443
2019-01-07T08:11:37.684106+00:00 app[web.1]: :reason_phrase => "Forbidden"
2019-01-07T08:11:37.684108+00:00 app[web.1]: :remote_ip     => "*******"
2019-01-07T08:11:37.684110+00:00 app[web.1]: :status        => 403
2019-01-07T08:11:37.684111+00:00 app[web.1]: :status_line   => "HTTP/1.1 403 Forbidden\r\n"
2019-01-07T08:11:37.684113+00:00 app[web.1]: ):
2019-01-07T08:11:37.684217+00:00 app[web.1]: F, [2019-01-07T08:11:37.684147 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737]
2019-01-07T08:11:37.684392+00:00 app[web.1]: F, [2019-01-07T08:11:37.684328 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737] app/controllers/microposts_controller.rb:7:in `create'

---

Bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::***********:user/user-name"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name/*",
                "arn:aws:s3:::bucket-name"
            ]
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::*******:user/user-name"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::bucket-name",
            "Condition": {}
        }
    ]
}

---

CORS:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

---

Current Initializer: carrier_wave.rb

if Rails.env.production?
  CarrierWave.configure do |config|
    config.fog_credentials = {
      # Configuration for Amazon S3
      :provider              => 'AWS',
      :aws_access_key_id     => ENV['S3_ACCESS_KEY'],
      :aws_secret_access_key => ENV['S3_SECRET_KEY'],
      :region                => ENV['S3_REGION']
    }
    config.fog_directory     =  ENV['S3_BUCKET']
  end
end

---

picture_uploader.rb

class PictureUploader < CarrierWave::Uploader::Base
  include CarrierWave::MiniMagick
  process resize_to_limit: [400, 400]

  if Rails.env.production?
    storage :fog
  else
    storage :file
  end

  # Override the directory where uploaded files will be stored.
  # This is a sensible default for uploaders that are meant to be mounted:
  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  # Add a white list of extensions which are allowed to be uploaded.
  def extension_whitelist
    %w(jpg jpeg gif png)
  end
end

Answer 1

Just had this problem and found out for private buckets with CarrierWave and fog you need to add this config.fog_public = false to your CarrierWave.configure .

After setting fog_public to false, the urls returned will be ones with signature too.

Docs: https://www.rubydoc.info/gems/carrierwave/CarrierWave/Storage/Fog

Answer 2

All tutorials I saw on how to upload a file in S3 using Carrierwave/fog had enabled Public access on their buckets. I had not for obvious reasons and so needed the config.aws_acl = :private in my carrierwave.rb config file. I think this will help others who face the same problem after following online tutorials trying to get this to work.