AWS Kinesis Firehose：如何使用 aws cli 和 bash 放置包含 JSON 的多个文件

Question

I have >100 files where each line is a json. It looks something like this (no commas & no []):

{"one":"one","two":{"tree":...}}
{"one":"one","two":{"tree":...}}
...
{"one":"one","two":{"tree":...}}

To be able to use aws firehose put-record-batch, file needs to be in the format:

[
  {
    "Data": blob
  },
  {
    "Data": blob
  },
  ...
]

I want to put all of these files to aws Firehose from terminal.

I'm looking to write a shell script that looks something like this:

for file in files
do
  aws firehose put-record-batch --delivery-stream-name <name> --records file://$file
done

So there're 2 questions:

1. How to transform the files into the applicable format
2. And, how to iterate through all the files

Answer 1

for file in *.json;
do
    jq -s . "${file}" >${file}.tmp && mv ${file}.tmp $file    
done

This will read all the json file in the current directory and change it into the desired form and save to the file .

OR if you do not have jq here is alternate way using python's json module.

for file in *.json;do
  while read line ; do 
      echo $line | python -m json.tool 
  done < ${file} |awk 'BEGIN{print "["}{print}END{print "]"}'
done