jq：使用输入文件有条件地更新/替换/添加json元素

Question

我收到以下输入文件：

• input.json：

[
 {"ID":"aaa_12301248","time_CET":"00:00:00","VALUE":10,"FLAG":"0"},
 {"ID":"aaa_12301248","time_CET":"00:15:00","VALUE":18,"FLAG":"0"},
 {"ID":"aaa_12301248","time_CET":"00:30:00","VALUE":160,"FLAG":"0"},

 {"ID":"bbb_0021122","time_CET":"00:00:00","VALUE":null,"FLAG":"?"},
 {"ID":"bbb_0021122","time_CET":"00:15:00","VALUE":null,"FLAG":"?"},
 {"ID":"bbb_0021122","time_CET":"00:30:00","VALUE":22,"FLAG":"0"},

 {"ID":"ccc_0021122","time_CET":"00:00:00","VALUE":null,"FLAG":"?"},
 {"ID":"ccc_0021122","time_CET":"00:15:00","VALUE":null,"FLAG":"?"},
 {"ID":"ccc_0021122","time_CET":"00:30:00","VALUE":20,"FLAG":"0"},

 {"ID":"ddd_122455","time_CET":"00:00:00","VALUE":null,"FLAG":"?"},
 {"ID":"ddd_122455","time_CET":"00:15:00","VALUE":null,"FLAG":"?"},
 {"ID":"ddd_122455","time_CET":"00:30:00","VALUE":null,"FLAG":"?"},
]

如您所见，有一些有效值（FLAG：0）和一些无效值（FLAG：“？”）。 现在，我得到了一个看起来像这样的文件（每个ID一个）：

aaa.json：

[
  {"ID":"aaa_12301248","time_CET":"00:00:00","VALUE":10,"FLAG":"0"},
  {"ID":"aaa_12301248","time_CET":"00:15:00","VALUE":null,"FLAG":"?"},
  {"ID":"aaa_12301248","time_CET":"00:55:00","VALUE":45,"FLAG":"0"}
]

如您所见，对象1与input.json中的对象相同，但是对象2无效（FLAG：“？”）。 这就是为什么必须用input.json（值：18）中的正确对象替换对象2的原因。 可以通过“ time_CET”和“ ID”元素来标识对象。

另外，input.json中将有新对象，这些对象不是aaa.json等的一部分。应将这些对象添加到数组，并应保留aaa.json中的有效对象。

最后，aaa.json应该如下所示：

[
  {"ID":"aaa_12301248","time_CET":"00:00:00","VALUE":10,"FLAG":"0"},
  {"ID":"aaa_12301248","time_CET":"00:15:00","VALUE":18,"FLAG":"0"},
  {"ID":"aaa_12301248","time_CET":"00:30:00","VALUE":160,"FLAG":"0"},
  {"ID":"aaa_12301248","time_CET":"00:55:00","VALUE":45,"FLAG":"0"}
]

因此，总结一下：

1. 寻找旗帜：“？” 在aaa.json中
2. 将此对象替换为使用“ ID”和“ time_CET”进行映射的input.json中的匹配对象。
3. 继续存在有效的对象，并从aaa.json之前添加的input.json中添加对象（这意味着仅在“ ID”字段中以“ aaa”开头的对象）
4. 对bbb.json，ccc.json和ddd.json重复此步骤

我不确定是否可以使用这样的命令一次完成所有操作，因为输出必须返回到正确的id文件（aaa，bbb ccc.json）：

jq --argfile aaa aaa.json --argfile bbb bbb.json .... -f prog.jq input.json

问题在于，标识符（aaa，bbb，ccc等）之后的数字可能会更改。 因此，要确保将对象添加到正确的文件/数组，将需要这样的语句：
if (."ID"|contains("aaa")) then ....

或者使用不同的输入参数多次运行该程序会更好吗？ 我不确定..

先感谢您！！

Answer 1

这是一种方法

#!/bin/bash

# usage: update.sh input.json aaa.json bbb.json....
# updates each of aaa.json bbb.json.... 

input_json="$1"
shift

for i in "$@"; do
    jq -M --argfile input_json "$input_json" '

      # functions to restrict input.json to keys of current xxx.json file
      def prefix:              input_filename | split(".")[0];
      def selectprefix:        select(.ID | startswith(prefix));

      # functions to build and probe a lookup table
      def pk:                  [.ID, .time_CET];
      def lookup($t;$k):       $t | getpath($k);
      def lookup($t):          lookup($t;pk);
      def organize(s):         reduce s as $r ({}; setpath($r|pk; $r));

      # functions to identify objects in input.json missing from xxx.json
      def pks:                 paths | select(length==2);
      def missing($t1;$t2):    [$t1|pks] - [$t2|pks] | .[];
      def getmissing($t1;$t2): [ missing($t1;$t2) as $p | lookup($t1;$p)];

      # main routine
        organize(.[]) as $xxx
      | organize($input_json[] | selectprefix) as $inp
      | map(if .FLAG != "?" then . else . += lookup($inp) end)
      | . + getmissing($inp;$xxx)

    ' "$i" | sponge "$i"

done

该脚本在循环中使用jq读取和更新每个aaa.json ...文件。

该过滤器创建临时对象，以方便查找值[ID,time_CET]将更新的任何值aaa.json与FLAG ==“？” 最后添加aaa.json input.json中缺少的所有值。

用于input.json的临时查找表使用input_filename以便仅包含以与当前处理的文件的名称匹配的前缀开头的键。

样品运行：

$ ./update.sh input.json aaa.json

aaa.json后的aaa.json ：

[
  {
    "ID": "aaa_12301248",
    "time_CET": "00:00:00",
    "VALUE": 10,
    "FLAG": "0"
  },
  {
    "ID": "aaa_12301248",
    "time_CET": "00:15:00",
    "VALUE": 18,
    "FLAG": "0"
  },
  {
    "ID": "aaa_12301248",
    "time_CET": "00:55:00",
    "VALUE": 45,
    "FLAG": "0"
  },
  {
    "ID": "aaa_12301248",
    "time_CET": "00:30:00",
    "VALUE": 160,
    "FLAG": "0"
  }
]