S3 on EC2 with IAM: Error NoCredentialProviders: no valid providers in chain. Deprecated

Question

My application use s3 and running on EC2. The IAM is configured on the instance, so the auth happen keyless (without the access key and secret key).

I'm able to upload or download file using aws cli. However when I tried to perform download operation using aws-sdk-go, I get error below:

AccessDenied: Access Denied
status code: 403, request id: F945BDB5410E1A00, host id: m74jJ8z/AEzdkaJkWKdIqPEwPIYPZfWnLLfa5UpEwHwaBcXOuXTPY1aw/u/5HGralKg+ewAWEJA=

I followed the official guide from https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/ and from this issue https://github.com/aws/aws-sdk-go/issues/430 but got the error above.

Below is my code:

s3UploadPath = config.GetString("upload assets to s3.bucket")
s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)

session, err := session.NewSession(s3Config)
if err != nil {
    Logger.Fatal("Error initializing s3 uploader. " + err.Error())
    os.Exit(0)
}

// the upload code
uploader = s3manager.NewUploader(session)
res, err := uploader.Upload(&s3manager.UploadInput{
    Bucket: aws.String(s3UploadPath),
    Key:    aws.String(filename),
    Body:   f,
})
if err != nil {
    log.Fatal("error on upload. " + err.Error())
}

// then continue with the download code

Attached screenshot showing that the download and upload operations are success through aws cli

Am I doing it wrong?

Answer 1

You dont need to specify credentials when using IAM role on EC2 instance.

I see you are getting Access Denied which means your Go program is able to pick the EC2 profile creds but probably due to lack of permissions, its getting this error.

Reading your code, it seems you want to write object to S3. Can you make sure you have given s3:Get* , s3:List* , s3:PutObject , s3:PutObjectAcl to your IAM Role and there is no explicit Deny on S3 Bucket policy?

Answer 2

I managed to solve this error by doing two things.

The first one is by using stscreds.NewCredentials(session, roleArn) as the credentials during session creation.

s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)
s3Config.WithLogLevel(aws.LogDebugWithHTTPBody)
s3Config.Region = aws.String(region)
s3Config.WithHTTPClient(&http.Client{
    Transport: &http.Transport{
        Proxy: http.ProxyFromEnvironment,
    },
    Timeout: 10 * time.Second,
})

sess, err := session.NewSession(s3Config)
if err != nil {
    log.Fatal("Error initializing session. " + err.Error())
}

sess.Config.Credentials = stscreds.NewCredentials(sess, arn)
_, err = sess.Config.Credentials.Get()
if err != nil {
    log.Fatal("Error getting role. " + err.Error())
}

And the 2nd thing is by defining NO_PROXY environment variable with value is 169.254.169.254 . The particular IP is AWS global IP used for getting the EC2 metadata.

And since my application uses proxy to communicate with the S3 server, I need to exclude that IP.