如何使用 wget 从 S3 存储桶下载文件？

Question

I can push some content to an S3 bucket with my credentials through S3cmd tool with s3cmd put contentfile S3://test_bucket/test_file

Question is, I am required to download the content from this bucket in other computers that don't have s3cmd installed on them, BUT they have wget installed.

when I try to download some content from my bucket with wget I get this:

 https://s3.amazonaws.com/test_bucket/test_file
--2013-08-14 18:17:40--  `https`://s3.amazonaws.com/test_bucket/test_file
Resolving s3.amazonaws.com (s3.amazonaws.com)... [ip_here]
Connecting to s3.amazonaws.com (s3.amazonaws.com)|ip_here|:port... connected.
HTTP request sent, awaiting response... 403 Forbidden
`2013`-08-14 18:17:40 ERROR 403: Forbidden.

I have manually made this bucket public through the Amazon AWS web console.

Question is : How can I download content from an S3 bucket with wget? into a txt local file?

Answer 1

You should be able to access it from a url created as follows:

http://{bucket-name}.s3.amazonaws.com/<path-to-file>

Now, say your s3 file path is:

s3://test-bucket/test-folder/test-file.txt

You should be able to wget this file with following url:

http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt

Answer 2

1. Go to S3 console

2. Select your object

3. Click 'Object Actions'

4. Choose 'Download As'

5. Use your mouse right-click to 'Copy Link Address'

6. Then use the command:

   wget --no-check-certificate --no-proxy 'http://your_bucket.s3.amazonaws.com/your-copied-link-address.jpg'

Answer 3

AWS cli has a 'presign' command that one can use to get a temporary public URL to a private s3 resource.

aws s3 presign s3://private_resource

You can then use wget to download the resource using the presigned URL.

Answer 4

Got it ... If you upload a file in an S3 bucket with S3CMD with the --acl public flag then one shall be able to download the file from S3 with wget easily ...

Conclusion: In order to download with wget, first of one needs to upload the content in S3 with s3cmd put --acl public --guess-mime-type <test_file> s3://test_bucket/test_file

alternatively you can try:

s3cmd setacl --acl-public --guess-mime-type s3://test_bucket/test_file

notice the setacl flag above. THAT WILL set the file in s3 accessible publicly then you can execute the wget http://s3.amazonaws.com/test_bucket/test_file

Answer 5

I had the same situation for couple of times. It's the fastest and the easiest way to download any file from AWS using CLI is next command:

aws s3 cp s3://bucket/dump.zip dump.zip

File downloaded way faster than via wget, at least if you are outside of US.

Answer 6

I had the same error and I solved it by adding a Security Groups Inbound rule:

HTTPS type at port 443 to my IP address ( as I'm the only one accessing it ) for the subnet my instance was in.

Hope it helps anyone who forgot to include this

Answer 7

Please make sure that the read permission has been given correctly.

If you do not want to enter any account/password, just by wget command without any password, make sure the permission is like the following setting shows.

By Amazon S3 -> Buckets -> Permisions - Edit Check the Object for "Everyone (public access)" and save changes. permission setting like this - screenshot

or choose the objest and go to "Actions" -> "Make public", would do the same thing under permission settings.

Answer 8

incase you do not have access to install aws client on ur Linux machine try below method.

• got to the bucket and click on download as button. copy the link generated.

• execute command below

  wget --no-check-certificate --no-proxy --user=username --ask-password -O "download url"

Thanks

Answer 9

you have made the bucket public, you need to also make the object public. also, the wget command doesn't work with the S3:// address, you need to find the object's URL in AWS web console.

Answer 10

I know I'm too late to this post. But thought I'll add something no one mentioned here.

If you're creating a presigned s3 URL for wget, make sure you're running aws cli v2. I ran into the same issue and realized s3 had this problem

Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4

This gets resolved once you presign on aws cli v2

Answer 11

The simplest way to do that is to disable Block all public firstly.

1. Hit your bucket name >> go to Permissions >> Block public access (bucket settings)

2. If it is on >> hit Edit >> Uncheck the box, then click on Save changes

3. Now hit the object name >> Object action >> Make public using ACL >> then confirm Make public

4. After that, copy the Object URL , and proceed to download

I hope it helps the future askers. Cheers

Answer 12

I had the same mistake

I did the following :

1. created IAM role > AWS Service type > AmazonS3FullAccess policy inside
2. applied this role to the EC2 instance
3. in the the Security Groups opened inbound HTTP and HTTPS to Anywhere-IPv4
4. made the S3 bucket public
5. profit! wget works! ✅