[英]Combining JSON with different Key Names
EDIT: Going to try and simplify my question, and the JSON examples to just relevant elements.编辑:尝试简化我的问题,并将 JSON 示例简化为相关元素。
Building a playbook in Ansible, and one task I am trying to do involves pulling down data from 4 separate Qradar API endpoints, and trying to combine some details from each of the endpoints.在 Ansible 中构建剧本,我正在尝试执行的一项任务涉及从 4 个单独的 Qradar API 端点提取数据,并尝试组合来自每个端点的一些细节。
4 different json sources for each of the endpoints:每个端点有 4 个不同的 json 源:
Examples from Lab:来自实验室的例子:
regex_properties.json
[
{
"identifier": "59723052-d96c-4cef-ba7b-69d426602e04",
"property_type": "numeric",
"name": "indexTotalSize",
}
]
log_sources.json
[
{
"id": 64,
"name": "SIM Audit-2 :: eng-qradar-aio-01",
"type_id": 105,
"last_event_time": 1588628234930,
}
]
log_source_types.json
[
"name": "SIM Audit",
"id": 105
},
]
property_expressions.json
[
{
"identifier": "0311c65b-d5b5-483e-943f-b539543a8e95",
"log_source_type_id": 105,
"log_source_id": 65,
"regex_property_identifier": "59723052-d96c-4cef-ba7b-69d426602e04",
}
]
I would like to pull in these 4 sources, and output a file that has the following data linkeed by the property_expressions.json:我想提取这 4 个来源,以及 output 一个文件,该文件具有由 property_expressions.json 链接的以下数据:
Such as below比如下面
merged_example.json
[
{
"identifier": "0311c65b-d5b5-483e-943f-b539543a8e95",
"log_source_type_id": 105,
"log_source_id": 65,
"regex_property_identifier": "59723052-d96c-4cef-ba7b-69d426602e04",
"property_type": "numeric",
"regex_name": "indexTotalSize",
"lst_name": "SIM Audit",
"ls_name": "SIM Audit-2 :: eng-qradar-aio-01",
}
]
Or into a csv with the same data, which the end goal of the export, but can wait.或者换成一个csv 用同样的数据,里面的目的是导出,但是可以等待。
I tried to rename "identifier" to "regex_property_identifier" in regex_properties.json, then using 'jq -s regex_properties.json property_expressions.json' but I'm still just seeing both contents being separate arrays in the same output/file. I tried to rename "identifier" to "regex_property_identifier" in regex_properties.json, then using 'jq -s regex_properties.json property_expressions.json' but I'm still just seeing both contents being separate arrays in the same output/file.
I've tried using ansible and doing something like:我尝试使用 ansible 并执行以下操作:
- name: use JQ to reformat json to csv
shell: cat /tmp/property_expressions.json | jq -r '.[]' | jq 'select(.enabled == true)' | jq '[.identifier,.regex_property_identifier,.log_source_id,.log_source_type_id] | @csv' > /tmp/props.csv
- name: Read CSV into dictionary
read_csv:
path: "/tmp/props.csv"
fieldnames: "prop_id,regex_id,ls_id,lst_id"
delimiter: ","
register: props
- name: Loop Prop Dictionary and replace in CSV the regex_id
replace:
path: "/tmp/props.csv"
regexp: "{{ item.regex_id }}"
replace: "{{ regex_properties.json | json_query(regex_name_q) }},{{ regex_properties.json | json_query(regex_type_q) }}"
loop: "{{ props.list }}"
vars:
regex_name_q: "{{ item.regex_id }}.name"
regex_type_q: "{{ item.regex_id }}.property_type"
In order to just make a CSV and find/replace the terms item by item.为了只制作 CSV 并逐项查找/替换条款。 But if I can do it within the JSON arrays that'd be cleaner.
但是,如果我可以在 JSON arrays 内做到这一点,那会更干净。
Assuming away the minor errors in the JSON examples in the question, the following bash script produces the output as shown:假设消除问题中 JSON 示例中的小错误,以下 bash 脚本会生成 output,如下所示:
#!/bin/bash
jq -n \
--argfile lst log_source_types.json \
--argfile ls log_sources.json \
--argfile pe property_expressions.json \
--argfile rp regex_properties.json '
[ range(0, $pe|length) as $i
| {identifier: $pe[$i].identifier,
log_source_type_id: $lst[$i].id,
log_source_id: $pe[$i].log_source_id,
regex_property_identifier: $pe[$i].regex_property_identifier,
property_type: $rp[$i].property_type,
regex_name: $rp[$i].name,
lst_name: $lst[$i].name,
ls_name: $ls[$i].name
}
]
'
Note: I wouldn't be too concerned that --argfile is officially deprecated, but if that bothers you, there are many workarounds, though some are version-dependent.注意:我不会太担心 --argfile 已被正式弃用,但如果这让您感到困扰,有很多解决方法,尽管有些是版本相关的。 If you want a non-deprecated solution that will work with every version of jq, I'd use the form:
如果您想要一个适用于每个 jq 版本的非弃用解决方案,我会使用以下形式:
jq -s -f program.jq \
log_source_types.json \
log_sources.json \
property_expressions.json \
regex_properties.json
where program.jq begins by defining the four $-variables, beginning with: .[0] as $lst |
其中 program.jq 首先定义四个 $ 变量,以:
.[0] as $lst |
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.