AWS find max file size in S3 bucket

Question

I want to find the size and name of the biggest file in my S3 bucket.

Currently I have:

aws s3api list-objects --bucket bucket-name --output json --query "[max(Contents[].Size), length(Contents[])]"

which does not allow me to see the name of the file.

I also have the command to list the details of all files on the bucket:

aws s3api list-object-versions --bucket bucket-name --query 'Versions[*].Size'

What command will give me the name and size of the largest file(s) on the S3 bucket?

Answer 1

Using AWS CLI only, this will find the largest file:

aws s3api list-objects-v2 --bucket bucket-name --query "sort_by(Contents, &Size)[-1:]"

or to include non-current versions if applicable:

aws s3api list-object-versions --bucket bucket-name --query "sort_by(Versions[*], &Size)[-1:]"

Optional tweaks:

• Replace -1 with -N to find the largest N files.
• Include .[Key,Size] at the end of the --query to select only those fields.

Sadly I think the filtering is done client side because this downloaded 28 MB when run on a large bucket. However it is still a useful 1-liner despite not being quick.

Answer 2

The following should return the name and size of the largest file in the bucket "bucket-name".

aws s3api list-object-versions --bucket bucket-name | jq -r '.Versions[] | "\(.Key)\t \(.Size)"' | sort -k2 -r -n | head -1

The command above uses jq which you can install from https://stedolan.github.io/jq/download/

Answer 3

Here is what I did:

using ">" operator sent the output of your command to a sizes.txt file. Then searched the max size in that text file to find the corresponding filename.

steps:

touch sizes.txt
aws s3api list-object-versions --bucket bucket-name | jq -r '.Versions[] | "\(.Key)\t \(.Size)"' | sort -k2 -r -n > sizes.txt 
vi sizes.txt /"max_size_retrieved_from_command"