Print files in S3 matching some specific pattern

Question

I have a S3 bucket, in the bucket i have one folder and inside the folder i have many sub folders like mentioned below.

Bucket path: s3://path1/path2

Inside path2 folders will be like

D1D04021200040609001
D1D04021200040612001
D3D04020000040603001
D3D04020000040606001
D6D05091200051512001
D6D05091200051518001
G1S05101200051217001
G1S05101200051218001
G4S05091200051012001
G4S05091200051013001

I have many folders like this. I just want to check specific file exists or not using a pattern. How can i do this in linux

aws s3api list-objects --bucket path1/path2 --query "Contents[?contains(Key, `D1D0`)]"

am getting error like.... invalid bucket name bucket name should be in regex

I ran the below query

 aws s3api list-objects --bucket bucket name --prefix daily/ --query "Contents[?contains(Key, 'A1D0518')]" --query "sum(Contents[].Size)" --output text

But i wonder yi am getting 10 lines of output... i need the total size of the files staring with A1D0518 enter image description here

Kindly help!

Answer 1

It appears that your requirement is to calculate the total size of all objects that contain a given string .

This can be done with:

 aws s3api list-objects --bucket my-bucket --query "sum(Contents[?contains(Key, 'E4Y0')].Size)"

If, instead, you are looking for the total size of all objects that start with a given string , use:

 aws s3api list-objects --bucket my-bucket --prefix E4Y0 --query "sum(Contents[].Size)"

Please note that Prefix looks at the full path, so it might include subdirectories. So, first have it print the names with --query Contents[].Key to make sure you are including the correct objects.

Update based on your comment:

To find objects in a given directory that being with E4Y0 use:

aws s3api list-objects --bucket my-bucket --prefix path1/path2/path3/E4Y0 --query "sum(Contents[].Size)"

But test it first by using this to list the objects being selected:

aws s3api list-objects --bucket my-bucket --prefix path1/path2/path3/E4Y0 --query "Contents[].Key"