Grep在S3文件夹中对文件进行排序

Question

I'd like to sort files in S3 folders and then check if files contain a certain string.

When I usually want to grep a file I do the following:

aws s3 cp s3://s3bucket/location/file.csv.gz - | zcat | grep 'string_to_find' 

I see I can sort files like this:

aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))'

Tried something like this so far but got broken pipe:

aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))' | cp - | zcat | grep 'string_to_find'

Answer 1

You can specify which fields to output and force them into text-only:

aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))[].[Key]' \
--output text

Basically, the sort_by and reverse output the Contents array, and this extracts the Key element. I put [Key] in square brackets to force each result onto its own line.