jq 1.5 - Change existing element or add new if does not exist

Question

Using:

• jq-1.5-1-a5b5cbe (Ubuntu 17.10)
• Compose the queries: https://jqplay.org/

Goals and conditions:

1. Replace an child object value with another value, at any depth, having parents objects or arrays, for example:
   • if .spec.template.spec.containers[ n ].env[ n ] .name == "CHANGEME" then
   • .spec.template.spec.containers[ n ].env[ n ] .value = "xx"
   • where n >=0
2. If any of the parents of .name do not exist, should be able to add them on the fly instead of exiting with an error
3. The output JSON should have at least the same elements as the input JSON, no existing elements should be lost
4. No duplicates are allowed within the elements of an array, but the order must be preserved , so functions like unique cannot be used

Sample input JSON:

The structure is actually imposed, so I have to obey it. An object "path" usually is something like: .spec.template.spec.containers[0].spec.env[1].name . You could also have .containers[1], and so on. This is highly variable, and sometimes some elements could exist or not, depends on a schema definition of that particular JSON.

[
  {
    "kind": "StatefulSet",
    "spec": {
      "serviceName": "cassandra",
      "template": {
        "spec": {
          "containers": [
            {
              "name": "cassandra",
              "env": [
                {
                  "name": "CASSANDRA_SEEDS",
                  "value": "cassandra-0.cassandra.kong.svc.cluster.local"
                },
                {
                  "name": "CHANGEME",
                  "value": "K8"
                }
              ]
            }
          ]
        }
      }
    }
  }
]

Scenarios

1. Replace an existing value while preserving the input structure, works as expected:
   • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[].env[] | select(.name==$v.name))|=$v)'

2. Let's assume I want to do the same, only that .env1 is the parent array of the object {name:"",value:""}. The expected output should be:

    [ { "kind": "StatefulSet", "spec": { "serviceName": "cassandra", "template": { "spec": { "containers": [ { "name": "cassandra", "env": [ { "name": "CASSANDRA_SEEDS", "value": "cassandra-0.cassandra.kong.svc.cluster.local" }, { "name": "CHANGEME", "value": "K8" } ], "env1": [ { "name": "CHANGEME", "value": "xx" } ] } ] } } } } ]

   • For this, I have tried to add an object env1 on the fly:
     • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[] | if .env1 == null then .+={env1:[$v]} | .env1 else .env1 end | .[] | select(.name==$v.name))|=$v)'
       • works if .env1 exists, else:
       • error: Invalid path expression near attempt to access element "env1" of {"name":"cassandra","env"..
       • same results if using notations like .env//[$v] or .env//=.env[$v]
     • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[].env1 | .[if length<0 then 0 else length end]) |= $v)'
       • works if .env1 does not exist
       • adds another element if the array .env1 exists, potentially duplicating objects
   • Eventually I have managed to create a working filter:
     • jq -r 'def defarr: if length<=0 then .[0] else .[] end; def defarr(item): if length<=0 then .[0] else foreach .[] as $item ([]; if $item.name == item then $item else empty end; .) end; map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec | .containers1 | defarr | .env1 | defarr($v.name) ) |=$v)'
       • this works as expected, however is too long and heavy and have to add the custom functions after each potential array in the object hierarchy

The question

Is there any way to simplify all this, make it a bit more generic to handle any number of parents, arrays or not?

Thank you.

Answer 1

"The question"

In answer to The question : Yes. jq 1.5 has keys_unsorted , so you can use the following def of walk/1 , which is now standard in the “master” version of jq:

# Apply f to composite entities recursively, and to atoms
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys_unsorted[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f
  end;

For further details and examples, see the “development” version of the jq manual, the jq FAQ https://github.com/stedolan/jq/wiki/FAQ , etc.

"No duplicates are allowed within the elements of an array"

This is readily accomplished using index/1 ; you might like to use a helper function such as:

def ensure_has($x): if index([$x]) then . else . + [$x] end;

"If any of the parents of .name do not exist, should be able to add them on the fly"

If I understand this requirement correctly, it will useful for you to know that jq will create objects based on assignments, eg

{} | .a.b.c = 1

yields

{"a":{"b":{"c":1}}}

Thus, using your example, you will probably want to include something like this in your walk :

if type == "object" and has("spec")
   then (.spec.template.spec.containers? // null) as $existing
   | if $existing then .spec.template.spec.containers |= ... 
     else .spec.template.spec.containers = ...
     end
else .
end

Answer 2

Managed to reach a very good form:

1. Added the following functions in ~/.jq :

    def arr: if length<=0 then .[0] else .[] end; def arr(f): if length<=0 then .[0] else .[]|select(f) end//.[length]; def when(COND; ACTION): if COND? // null then ACTION else . end; # Apply f to composite entities recursively, and to atoms def walk(f): . as $in | if type == "object" then reduce keys_unsorted[] as $key ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f elif type == "array" then map( walk(f) ) | f else f end; def updobj(f): walk(when(type=="object"; f));

2. A typical filter will look like this:

    jq -r '{name:"CHANGEME",value: "xx"} as $v | map( when(.kind == "StatefulSet"; .spec.template.spec.containers|arr|.env|arr(.name==$v.name)) |= $v)'

The result will be that all objects that do not exist already will be created. The convention here is to use the arr functions for each object that you want to be an array, and at the end use a boolean condition and an object to replace the matched one or add to the parent array, if not matched.

3. If you know the path is always there, and so are the object you want to update, walk is more elegant:

    jq -r 'map(updobj(select(.name=="CHANGEME").value|="xx"))'

---

Thank you @peak for your effort and inspiring the solution.