有没有办法使用 aws s3 ls cli 将 S3 存储桶名称添加到存储桶的递归列表中？

Question

I prefer using the aws cli for listing S3 contents; it is handy for generating a file of object details that I can sort, grep, and otherwise manipulate later.

Unfortunately, by default it doesn't put the S3 bucket name in the object name. For example, if I want to list a bucket called example, I type and get this:

% aws s3 ls s3://example
2021-12-23 15:31:17     8572 object_name
2021-12-22 08:45:23       11 another_object_name

Is there a way to get the aws cli to put the bucket name on each line? Then I can grep across a file or files covering multiple buckets and see which bucket each object is in.

Like this:

% aws s3 ls s3://example
2021-12-23 15:31:17     8572 s3://example/object_name
2021-12-22 08:45:23       11 s3://example/another_object_name

I don't see an option in the AWS cli docs to do it, but perhaps someone knows an undocumented flag or something.

Answer 1

The AWS CLI offers two sub-commands to interact with S3. There is the high-level s3 sub-command you're using. This command allows very straight forward access to the most common actions on S3 buckets, but is limited in its functionality and doesn't expose all features of the underlying API.

The other sub-command is s3api , which offers direct access to the S3 API. With s3api you're quite flexible regarding the formatting of the output, as you can apply a JMESPath expression before returning it.

Here is an example which comes close to your desired output. It's not a perfect representation (note the difference in the date format and the alignment of the object sizes), but should be close enough:

$ BUCKET_NAME=example aws s3api list-objects-v2 --bucket $BUCKET_NAME \
    --query 'Contents[].[LastModified, Size, join(`/`, [`s3://'$BUCKET_NAME'`, Key])]' \
    --output text
2021-12-23T15:31:17.000Z        8572 s3://example/object_name
2021-12-22T08:45:23.000Z        11   s3://example/another_object_name

Answer 2

I decided it would be easiest to do this in the shell.

That means I used this:

aws s3 ls | sed s/....................//|sed 's#.*#BN=&; aws s3 ls s3://& --recursive | sed "s,  , ,g;s,  , ,g;s,  , ,g;s,  , ,g"|sed "s, , s3://$BN/,3"#' | sh -x > filename