使用 bash、sed、grep 或 awk 從無效的 JSON 中提取數據？

Question

我正在嘗試在 bash 中解析無效的 JSON

x="{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"

使用以下腳本

for each in $(echo $x | sed 's/{componentId: /\n/g' ); do
    echo "Each: $each"
    echo [[ $each == 0Rb* ]]
    if [[ $each == 0Rb* ]]; then
        component=echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $3}'
        reference=echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $6}'
        echo "component: $component"
        echo "reference: $component"
    fi
done

但它不起作用。 我不明白為什么它不起作用。 當我在控制台中執行這一行時，

echo $x | sed 's/{componentId: /\n/g' 

我可以看到這個無效的 json 被正確地分成幾行，但是當我嘗試將它傳遞給 for-loop 時，每個變量都會將更小的塊接收到它的值中

Each: 00N5E000005vm9e,

我很迷惑。

我想要做的是提取componentName:和之間的值,以及referenceName:和之間的另一個值，當componentId不以00N開頭時,對於無效的 json 中的每個項目。 有沒有辦法做到這一點？

我也嘗試過使用jq -n $x但它失敗了jq: error: syntax error, unexpected IDENT, expecting '}' (Unix shell quoting issues?) at <top-level>, line 1:

Answer 1

將數據視為 JSON

使用 sed 將其轉換回有效的sed ，例如：

# Remove redundant space (assuming the text is in the `x` variable)
<<<"$x" sed 's/: /:/g; s/, /,/g' |

# Quote all "words"
sed -E 's/[^"{}:,]+/"&"/g'       |

# Separate objects
sed 's/},{/}\n{/g'               |

# Parse json
jq .

Output：

{
  "componentId": "00N5E000005vm9e",
  "componentName": "Field",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "00N5E000005vm9e",
  "componentName": "Field",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "00N5E000005vm9e",
  "componentName": "Field",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "0Rb5E000000BGVi",
  "componentName": "Versions",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "0Rb5E000000BGVj",
  "componentName": "Approves",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "0Rb5E000000BGVe",
  "componentName": "activityThreads",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "0Rb5E000000BGVf",
  "componentName": "Attachments",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}
{
  "componentId": "0Rb5E000000BGVh",
  "componentName": "Details",
  "referenceId": "0M05E0000002XbV",
  "referenceName": "RecordPageName1",
  "referenceUrl": "null",
  "message": "Component is in use by another component in your organization.",
  "reasonCode": "10"
}

要遍歷componentId和referenceId ，您可以使用 jq 的@tsv格式運算符，例如：

... | jq -r '[ .componentId, .referenceId ] | @tsv'

Output：

00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
0Rb5E000000BGVi 0M05E0000002XbV
0Rb5E000000BGVj 0M05E0000002XbV
0Rb5E000000BGVe 0M05E0000002XbV
0Rb5E000000BGVf 0M05E0000002XbV
0Rb5E000000BGVh 0M05E0000002XbV

將數據視為 YAML

正如@léa 所指出的，您可以使用yq將此字符串解析為 YAML 數組。 這是我使用Mike Farah 的 yq 4.13.2 版對這種方法的看法：

<<<"[$x]" yq e '.[] | .componentId + " " + .referenceId' -

Output：

00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
0Rb5E000000BGVi 0M05E0000002XbV
0Rb5E000000BGVj 0M05E0000002XbV
0Rb5E000000BGVe 0M05E0000002XbV
0Rb5E000000BGVf 0M05E0000002XbV
0Rb5E000000BGVh 0M05E0000002XbV

在 bash 循環中解析變量

您可以將上述解決方案的結果 pipe 用於while read循環，例如：

... | while read componentId referenceId; do 
  : Do your processing here with $componentId and $referenceId
done

Answer 2

此輸入字符串是 YAML 對象數組容器的一部分。 所以用 YAML 解析器解析它。

使用 Python：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
import yaml
import json

# Your input invalid JSON but valid YAML elements part of an array
x = "{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"

# Compose yamlstring from x by adding the missing data array container
yamlstring = "data: [" + x + "]"

# Load data from the yamlstring
data = yaml.load(yamlstring, yaml.SafeLoader)

# Output data as JSON
json.dump(data, sys.stdout, indent=2)

或者從 shell 使用yq作為解析器：

#!/usr/bin/env sh

x="{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"

yamlstring="data: [$x]"

printf %s "$yamlstring" | yq -I 4 -o json e '.' -

Answer 3

感謝您的評論，看來我已經弄清楚了。

echo $x | sed 's/{componentId: /\n/g' | while IFS=\n read -r each; do
    #echo "Each: $each"
    #echo [[ $each == 0Rb* ]]
    if [[ $each == 0Rb* ]]; then
        component=$(echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $3}')
        reference=$(echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $6}')
        echo "component: $component"
        echo "reference: $reference"
    fi
done