Error on copying AWS s3 bucket to Azure Storage using AZCopy

Question

I have been trying to copy a AWS S3 bucket to my Azure Storage container following the Microsoft document Amazon S3 to Azure Storage by using AzCopy

But the operation fails with error as below

failed to perform copy command due to error: cannot start job due to error: cannot list objects, Access Denied.

I have tried to copy by authentication through both Azure AD and also SAS token with Full Access privileges (racwdl) but get same issue.

I have verified that the AWS keys set are of admin access and also verified that both my AWS bucket and Azure Storage Containers are public.

What else could be wrong? Any inputs would be helpful.

Also sharing the terminal outputs on using AD authorization and SAS token authorization.

On AD Authorization:

INFO: Scanning... INFO: Authenticating to destination using Azure AD INFO: Instead of transferring from the 's3.amazonaws.com' URL, in this version of AzCopy we recommend you use a region-specific endpoint to transfer from one specific region. Eg s3.us-east-1.amazonaws.com or a virtual-hosted reference to a single bucket. INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

failed to perform copy command due to error: cannot start job due to error: cannot list objects, Access Denied.

Using SAS Token Authorization

INFO: Scanning... INFO: Instead of transferring from the 's3.amazonaws.com' URL, in this version of AzCopy we recommend you use a region-specific endpoint to transfer from one specific region. Eg s3.us-east-1.amazonaws.com or a virtual-hosted reference to a single bucket. INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists. INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

failed to perform copy command due to error: cannot start job due to error: cannot list objects, Access Denied.

Answer 1

I have managed to resolve this issue. Apparently the solution is to add the AWS credentials as environment variables as setting them on CLI as mentioned in the documents doesn't work.

Answer 2

As stated above, the powershell script "set AWS_ACCESS_KEY_ID=....." doesn't work as it doesn't add the system environment variables in windows, so the access is denied to S3. To add the environment variables in Windows do the following:

1. Press Windows Key + R then open "control sysdm.cpl"
2. In System Properties click on Advanced -> Environment Variables
3. Under System Variables add the variable name: AWS_ACCESS_KEY_ID and place your access key under the value.
4. Add another system variable "AWS_SECRET_ACCESS_KEY" and place in the AWS secret key.
5. Go back to powershell and you should be able to run azcopy and the permissions should work.
6. If the permissions still do not work, then it might be an issue with the permissions in AWS.