Can we use copy command to load data into redshift table from S3 using access key and secret key (Without with IAM Role)

Question

I have a requirement where I need to load data into Redshift table from S3. Files present in the S3 are in Parquet format. I don't have an IAM policy but have an S3 Access Key and Secret Key.

Things to note: Redshift and S3 are in two different AWS accounts however these accounts are in the same region.

What I have tried:

copy abc from 's3://us-publish-prod/customer/r_cust_geo_funcl_terr_addr_elig/' 
credentials 'aws_access_key_id=XXXXXXXXX;aws_secret_access_key=XXXXXXXXXXX'
FORMAT AS PARQUET;

Error: Invalid operation: COPY from this file format only accepts IAM_ROLE credentials 

Thanks in Advance.

Answer 1

According to COPY from columnar data formats - Amazon Redshift , it seems that loading data from Parquet format requires use of an IAM Role rather than IAM credentials:

COPY command credentials must be supplied using an AWS Identity and Access Management (IAM) role as an argument for the IAM_ROLE parameter or the CREDENTIALS parameter.

This would mean using either of:

CREDENTIALS 'aws_iam_role=arn:aws:iam::<aws-account-id>:role/<role-name>'
IAM_ROLE 'arn:aws:iam::<aws-account-id>:role/<role-name'

Thus, you need to use an IAM Role , even if the files were stored in your own AWS account.

You will therefore need to:

• Ask your admins to create an IAM Role in the account with Amazon Redshift with permissions to use GetObject and ListBucket on the S3 bucket that contains the source data (in the other account)
• Ask the admins of the 'other' account to add the IAM Role to the bucket policy where the files are being stored, so that the IAM Role has GetObject and ListBucket permissions (most likely identical to what they have granted to your IAM User)
• Authorize the Redshift cluster to use the IAM Role
• Modify your COPY command to use the IAM Role instead of credentials

See: Authorizing Amazon Redshift to access other AWS services on your behalf