insert bash variable into json using jq

Question

I am trying to fill JSON template with increasing counter to generate huge sample data set:

#!/bin/bash
for ((counter=1 ; counter<2 ;counter++ ))
do
  NUMBER=${counter}
  JSON=$(cat template.json | jq --arg NUMBER "$NUMBER" '.')
  echo $JSON
  #aws dynamodb batch-write-item --request-items "${JSON}"
done

My template.json looks like:

{
"My_Table":[{
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...        
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_D"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_E"},"name":{ ...
}]

}

Can I get any clue to insert bash variable into JSON template? I guess I am not doing correctly using jq here:

JSON=$(cat template.json | jq --arg NUMBER "$NUMBER" '.')

EDIT My desired output:

{
"My_Table":[{
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...        
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_C"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_C"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_D"},"name":{ ...
    "PutRequest":{"Item":{"type":{"S":"test-1-Type_E"},"name":{ ...
}]

}

Answer 1

Unfortunately there are several parts of the question that don't quite make sense, but I believe the following should get you on your way.

First, I'll assume your template is valid JSON:

{
    "My_Table": [
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":0}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":1}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":2}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":0}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":1}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":2}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":0}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_D"},"name":1}}},
    {"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_E"},"name":0}}}
  ]
}

Second, I'll assume you want the result along the lines of what is shown, not as described, but the following is written so you can easily adapt the code to the described problem. Specifically, the functionality for making the substitution is encapsulated in the following definition:

def resolve(s; value):
  .My_Table |= map(.PutRequest.Item.type.S |= 
     sub("-" + s + "-"; "-" + (value|tostring) + "-" ));

This is written using sub , the first argument of which must be a regex. So, to generate the desired output for a single substitution of "$NUMBER" by "1", one would write:

resolve("\\$NUMBER"; 1)

Since I'm not sure what your bash snippet is supposed to do exactly, I'll just suggest that you could use iteration within jq to achieve whatever result you require, rather than using bash iteration.

Answer 2

If the template is intended to be used with jq , it should be changed to work correctly, rather than trying to force jq to work with substandard input.

{
    "My_Table":[{
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...        
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_C"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_C"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_D"},"name":{ ...
        "PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_E"},"name":{ ...
    }]
}

Now something like your original attempt will work correctly. (Name your template template.jq to emphasize that it is not actually valid JSON.)

for ((counter=1 ; counter<2 ;counter++ ))
do
  JSON=$(jq -n -f template.jq --arg NUMBER "$counter")
  echo "$JSON"
  #aws dynamodb batch-write-item --request-items "${JSON}"
done