Looping over a list of keys to extract from a JSON file with jq

Question

I'm trying to extract a series of properties (named in an input file) in jq and getting error when I feed those from bash via a loop:

while read line; do echo $line; cat big.json | jq ".$line"; sleep 1; done < big.properties.service

cfg.keyload.service.count
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:

When i try to do it manually it works

$ line=cfg.keyload.service.count
$ echo $line
cfg.keyload.service.count
$ cat big.json | jq ".$line"
1

Is there any way to get it work in loop?

Here is example

cat >big.json <<EOF
{
  "cfg": {
    "keyload": {
      "backend": {
    "app": {
      "shutdown": {
        "timeout": "5s"
                  },
      "jmx": {
        "enable": true
                    }
           }
       }
     }
}
}
EOF

cat >big.properties.service <<EOF
cfg.keyload.backend.app.shutdown.timeout
cfg.keyload.backend.app.jmx.enable
cfg.keyload.backend.app.jmx.nonexistent
cfg.nonexistent
EOF

...output should be:

cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null

Answer 1

Immediate Issue - Invalid Input

The "invalid character" at hand here is almost certainly a carriage return. Use dos2unix to convert your input file to a proper UNIX text file, and your original code will work (albeit very inefficiently, rereading your whole big.json every time it wants to extract a single property).

Performant Implementation - Loop In JQ, Not Bash

Don't use a bash loop for this at all -- it's much more efficient to have jq do the looping.

Note the sub("\\r$"; "") used in this code to remove trailing carriage returns so it can accept input in DOS format.

jq -rR --argfile infile big.json '
sub("\r$"; "") as $keyname
| ($keyname | split(".")) as $pieces
| (reduce $pieces[] as $piece ($infile; .[$piece]?)) as $value
| ($keyname, ($value | tojson))
' <big.properties.service

properly emits as output, when given the inputs in the question:

cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null

Answer 2

Your properties file is effectively paths in the json that you want to retrieve values from. Convert them to paths that jq recognizes so you can get those values. Just make an array of keys that would need to be traversed. Be sure to read your properties file as raw input ( -R ) since it's not json, and use raw output ( -r ) to be able to output the paths as you want.

$ jq --argfile big big.json '
    ., (split(".") as $p | $big | getpath($p) | tojson)
' -Rr big.properties.service
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null