base64 encode and decode of aws command to retrieve some fields

Question

Could someone explain the below code please?

The below code is producing the expected results with username.

for row in $(echo "${es_eh}" | jq -r '.Events[] | @base64'); do
    echo "${row}" | base64 --decode | jq -r '.Username'

I didn't understand the purpose of doing base64 encode and then doing decode of the same string inside loop to retrieve username.

This is not working when I remove base64 encode and decode.

for row in $(echo "${es_eh}" | jq -r '.Events[]'); do
    echo "${row}"| jq -r '.Username'

es_eh contains the json output of the below aws command,

es_eh="$(aws cloudtrail --region us-east-1 lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=route53.amazonaws.com --max-items 50 --start-time "${start_date}" --end-time  "${end_date}" --output json)"

Answer 1

Without the encoding, the output of the first jq is more than one row. The loop iterates over the lines and fails, as none of them contains a valid JSON. With the | @base64 | @base64 , each subobject is encoded to a single row, inflated back to a full JSON object by base64 --decode .

To see the rows, try outputting $row before processing it.

Answer 2

When you use $( ) without quotes around it, the result gets split into "words", but the shell's definition of a "word" is almost never what you want (and certainly has nothing to do with the json entries you want it split into). This sort of thing is why you should almost never use unquoted expansions.

Converting the output entries to base64 makes them wordlike enough that shell word splitting actually does the right thing. But note: some base64 encoders split their output into lines, which would make each line be treated as a separate "word". If jq 's base64 encoding did this, this code would fail catastrophically on large events.

Answer 3

Transforming the for loop into a while loop should fix the problem:

while read -r row; do
    echo "${row}" | jq -r '.Username' 
done < <(echo "${es_eh}" | jq -c -r '.Events[]')

Note that in the outer jq , I used option -c to put output in a single ine.