How can I parse a Json file in linux using bash or jq for setting a results

Question

On my linux machine I am trying parse a 'Results.json' file. The two strings I wanted to search and grep from it are

1. "first_review": true
2. "firstReview": true

If any one of them (or both of them) are true, then I want to print Results as 'fail' and only if both are 'false' , then I want to print results as Pass.

How can I go about it correctly and achieve this, kindly suggest.

Here is the Json file (results.json)

{
  "scan-dir": "/var/local",
  "scan_time_ms": "20394 ms",
  "by": "a user",
  "project": "local",
  "date": "2019-06-30T10:48:07.270Z",
  "g_version": "2.2.16",
  "q_version": "1.0.112",
  "directories_scanned": 25,
  "files_scanned": 41,
  "packages_found": 3,
  "packages": [
    {
      "name": "flow",
      "version": "unknown",
      "path": "development_environment/flow",
      "source": "__init__.py",
      "file_path": "development_environment/flow/__init__.py",
      "analyzer": "package-finder",
      "license_files": [
        {
          "file": "development_environment/flow/__init__.py",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# -*- coding: utf-8 -*-\n#\n# ",
              "matches": null
            },
            {
              "text": "Licensed",
              "matches": "KEYWORD"
            },
            {
              "text": " to the ",
              "matches": null
            },
            {
              "text": "Apache Software Foundation (ASF) under one",
              "matches": "ACCEPTABLE"
            },
            {
              "text": "\n# or more contributor ",
              "matches": null
            },
            {
              "text": "license",
              "matches": "KEYWORD"
            },
            {
              "text": " agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " this file\n# to you under",
              "matches": null
            },
            {
              "text": " the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.",
              "matches": "Apache-2.0"
            }
          ],
          "license_types": [
            "Apache-2.0"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        },
        {
          "file": "development_environment/flow/base_flow.py",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# -*- coding: utf-8 -*-\n#\n# ",
              "matches": null
            },
            {
              "text": "Licensed",
              "matches": "KEYWORD"
            },
            {
              "text": " to the ",
              "matches": null
            },
            {
              "text": "Apache Software Foundation (ASF) under one",
              "matches": "ACCEPTABLE"
            },
            {
              "text": "\n# or more contributor ",
              "matches": null
            },
            {
              "text": "license",
              "matches": "KEYWORD"
            },
            {
              "text": " agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " this file\n# to you under",
              "matches": null
            },
            {
              "text": " the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.",
              "matches": "Apache-2.0"
            },
            {
              "text": "\n\n\"\"\"Base classes for flow and flowBag.\"\"\"\n\nfrom abc import ABCMeta, abstractmethod\n\n\nclass Baseflow(metaclass=ABCMeta):\n    \"\"\"\n    Base flow object that both the Simpleflow and flow inherit.\n    \"\"\"\n\n    @property\n    @abstractmethod\n    def flow_id(self):\n        \"\"\"\n        :return: the flow ID\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def task_ids(self):\n        \"\"\"\n        :return: A list of task IDs that are in this flow\n        :rtype: List[unicode]\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def full_filepath(self):\n        \"\"\"\n        :return: The absolute path to the file that contains this flow's definition\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def concurrency(self):\n        \"\"\"\n        :return: maximum number of tasks that can run simultaneously from this flow\n        :rtype: int\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def is_paused(self):\n        \"\"\"\n        :return: whether this flow is paused or not\n        :rtype: bool\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def pickle_id(self):\n        \"\"\"\n        :return: The pickle ID for this flow, if it has one. Otherwise None.\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError\n\n\nclass BaseflowBag:\n    \"\"\"\n    Base object that both the SimpleflowBag and flowBag inherit.\n    \"\"\"\n    @property\n    @abstractmethod\n    def flow_ids(self):\n        \"\"\"\n        :return: a list of flow IDs in this bag\n        :rtype: List[unicode]\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def get_flow(self, flow_id):\n        \"\"\"\n        :return: whether the task exists in this bag\n        :rtype: tzlflow.flow.base_flow.Baseflow\n        \"\"\"\n        raise NotImplementedError()",
              "matches": null
            }
          ],
          "license_types": [
            "Apache-2.0"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "Apache-2.0"
      ],
      "license_approval_flags": {
        "approved": true,
        "first_review": false,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Approved",
      "package_hash": "c7d6757c6c814a22b44d8299cace3ec1",
      "Treat": {
        "TreatReviewed": false,
        "possiblyRelatedPackages": [],
        "searchedName": "flow",
        "searchedVersion": "unknown",
        "allLicensesMatch": false,
        "firstReview": true,
        "firstReviewReason": "New package and not exempt"
      }
    },
    {
      "name": "testscan-results",
      "version": "unknown",
      "language": "unknown",
      "analyzer": "License verify",
      "path": "testscan-results",
      "file_path": "testscan-results/License-Report.html",
      "license_files": [
        {
          "file": "testscan-results/License-Report.html",
          "hash": "db02a6be21775d25af4cfdb993442a8e",
          "legal": true,
          "legal_category": "License",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "<style>h2 strong{color: red;} pre{padding-left:1em; background-color:cornsilk; border:1px solid black; margin: 0 2em}</style><h1 id=\"",
              "matches": null
            },
            {
              "text": "nolicenseswerefoundwhichrequirereview",
              "matches": "KEYWORD"
            },
            {
              "text": "\">No ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " were found which require review.</h1>",
              "matches": null
            }
          ],
          "license_types": [
            "KEYWORD",
            "UNKNOWN"
          ],
          "license_approval_flags": {
            "approved": false,
            "first_review": true,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        },
        {
          "file": "test_scan-results/License-Report.md",
          "hash": "b85fd1e259874c81c31253e352068b6f7",
          "legal": true,
          "legal_category": "License",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# No ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " were found which require review.",
              "matches": null
            }
          ],
          "license_types": [
            "KEYWORD",
            "UNKNOWN"
          ],
          "license_approval_flags": {
            "approved": false,
            "first_review": true,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "KEYWORD",
        "UNKNOWN"
      ],
      "license_approval_flags": {
        "approved": false,
        "first_review": true,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Needs Review",
      "package_hash": "11d905fbde2bbd890c5d3e677704185a"
    },
    {
      "name": "unknown",
      "version": "unknown",
      "language": "unknown",
      "analyzer": "License verify",
      "path": "development_environment/README.md",
      "file_path": "development_environment/README.md",
      "license_files": [
        {
          "file": "development_environment/README.md",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": false,
          "blocks": [
            {
              "text": "# tzlflow EnV\n\nA development environment for ",
              "matches": null
            },
            {
              "text": "Flow deploy upon tzl Cloud utilising shared Database i",
              "matches": "ACCEPTABLE"
            },
            {
              "text": ".e.",
              "matches": null
            },
            {
              "text": " PostGresql ",
              "matches": "PostgreSQL"
            },
            {
              "text": "and Messenging Services i.e. Redis.  Intention is that developers can get up and running quickly with working tzlflow environment (precanned to G+EHR software stack).\n\n## Development and Testing\nSetting up a development environment utilising developer free tzl Cloud. \n\n### Pre-requistes\n- Sign up for [tzl Cloud Account](",
              "matches": null
            },

            {
              "text": ")\n-- tzl Cloud CLI\n-- kubectl \n- Access to [repo](",
              "matches": null
            },
            {
              "text": ")\n- Docker\n- Helm\n\nFollowing two are not pre-req but you may decide to 'bring your own'\n- Postgres DB\n- Redis \nIf using your own following comment instructions in values.yaml to configure. \n\n### Clone repo\n\n- git clone  git@",
              "matches": null
            },

            {
              "text": ":TzlCo/Tzl-tools\n- cd Tzl-tools\n\n### Update configuration\n\nUnder",
              "matches": null
            },
            {
              "text": " postgresql ",
              "matches": "PostgreSQL"
            },

            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "port\n  service:\n    port: 30406\n\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "User to create.\n  postgresUser: tzl_cloud_xxx\n  ##\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "Password for the new user.\n  ## If not set, a random 10 characters password will be used.\n  postgresPassword: xxxx\n  ##\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "Database to create.\n  postgresDatabase: tzlclouddb\n```\n\nand Redis\n\n```redis:\n  ##\n  ## Use the redis chart dependency.\n  ## Set to false if bringing your own redis.\n  enabled: false\n  ##\n  ## If you are bringing your own redis, you can set the host in redisHost.\n  redisHost: xxxx.databases.appdomain.cloud\n  ##\n  ## Redis password\n  ##\n  password: xxxxx\n  username: admin\n  ##\n  ## Master configuration\n  master:\n    #Redis Port - missing from chart.\n    port: 31932\n    \n```\n\n### flow Git Repo configuration\n\nFollow instructions to generate ssh key for git repo.\n",
              "matches": null
            },

            {
              "text": "#deploy-key\nIf you are using a private Git repo, you can set `flows.gitSecret` to the name of a secret you created containing private keys and a `known_hosts` file.\n\nFor example, this will create a secret named `my-git-secret` from your ed25519 key and known_hosts file stored in your home directory:  `kubectl create secret generic flow-git-secret --from-file=gitSshKey=g+ehr_flows --from-file=known_hosts=known_hosts --from-file=gitSshKey.pub=g+ehr_flows.pub`\n\n\n\n###  Deploy\n\nLog onto tzl Cloud using CLI\n```tzlcloud login -a ",
              "matches": null
            },

            {
              "text": " -r us-south -g Tzl-Dev --sso```\n\nDownload the kubeconfig files for your cluster.\n```tzlcloud ks cluster-config --cluster Tzl_dev```\n\nUsing the output from the previous step, set the KUBECONFIG environment variable. The command looks similar to the following example:\n```export KUBECONFIG=/Users/$USER/.tzl/plugins/container-service/clusters/Tzl_dev/kube-config-dal10-Tzl_dev.yml```\n\nUsing Helm deploy tzlflow\n```helm install --namespace \"default\" --name \"tzlflow\" stable/tzlflow -f values.yaml```\n\nTreat access to tzlflow admin\n```export POD_NAME=$(kubectl get pods --namespace default -l \"component=web,app=tzlflow\" -o jsonpath=\"{.items[0].metadata.name}\")\n   echo http://127.0.0.1:8080\n   kubectl port-forward --namespace default $POD_NAME 8080:8080```\n   \n\n\n# TOD0 \n- python script to automate deploy. \n- Steps to integrate with different dependency e.g. postgres or redis.\n- flow script locations.\n- instructions to use github repo as flow location.\n",
              "matches": null
            }
          ],
          "license_types": [
            "PostgreSQL"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "PostgreSQL"
      ],
      "license_approval_flags": {
        "approved": true,
        "first_review": false,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Approved",
      "package_hash": "2776a8d434cd903a7cd441c9dcbdd4ed"
    }
  ],
  "Treat_checked": true
}

Here are the things I tried and but could not progres much further, on each of the approaches:

1. First Tried to used jq and could parse the json for the paths and get the values ie. 'true' or 'false' for the required nodes but dont know how to progress to next step, using those true and false values.

   used something like :

   cat results.json |jq '.packages[0].license_approval_flags.first_review'

2. In another thought wanted to Cat and then grep the json file contents, for above two required complete string values ("first_review": true and "firstReview": true) so that If get any results as 'true' then I wanted to set/print 'Results' as 'Fail', but could not get any correct responses as I think quotes to the strings is not being filtered in my grep search.

   used grep command as below:

   grep -o '"[firstReview = true"]\\+"' results.json

3. Then wanted to use Python to parse json and then get my desired work done but could not progress beyond this page 'how to parse data from json to python', which seems to detail something that I might use but I didnt yet have the python needed basics.

This probably is not tough as I read over internet but I am unable to complete this yet.

How can I go about it correctly and achieve this, please kindly suggest.

Answer 1

Here's a jq solution to the problem as I understand it. The solution is quite brief because it is agnostic about where the two keys are located, but you may need to modify it depending on your detailed requirements:

# Check if both are false
reduce (.. | objects) as $o ({};
  if $o.firstReview == false then .firstReview = false else . end
   | if $o.first_review == false then .first_review = false else . end )
| if length == 2 then "Pass" else "Fail" end

This solution is a tiny bit subtle because it relies on the semantics of length when applied to objects, but it has the possible advantage of robustness with respect to the locations of the keys of interest.

Per-package solution

If you want the results for each "package", simply add the wrapper .packages | map(_) .packages | map(_) , ie replace _ by the above filter, yielding:

.packages
| map(reduce (.. | objects) as $o ({};
      if $o.firstReview == false then .firstReview = false else . end
       | if $o.first_review == false then .first_review = false else . end )
    | if length == 2 then "Pass" else "Fail" end
)

Answer 2

If you want to just look for any named property in your json that has a true value, you have many options. For a quick and dirty solution, you could do this:

if any(..|objects|.first_review,.firstReview; . == true) then "Fail" else "Pass" end

Answer 3

you could do something like that:

cat results.json |
  jq '.packages[0].Treat.firstReview
       or .packages[0].license_approval_flags.first_review
      | if . == true then "Fail" else "Pass" end'

false

Answer 4

alternatively , the same could be achieved using a walk-path based unix utility jtc :

bash $ <results.json jtc -w'<r:"Pass">f <first_review>l <true>b >r:"Fail"<F <>f <firstReview>l <true>b <r:"Fail">v' -T'{{r}}' -qq
Fail
bash $ 

walk path ( -w ) elaboration:

• first define a fail-point (which will be a JSON root) and at the same time setup a namespace r set to "Pass" ( <r:"Pass">f )
• then walk the first branch of the path evaluating if first_review is set to boolean true ( <first_review>l <true>b ) and if it is then successfully end walking simultaneously rewriting namespace r to value '"Fail"' ( >r:"Fail"<F )
• if prior walk fails, then walk a second branch, also setting the fail stop ( <>f - empty this time) and again try evaluating firstReview the same way.

One walking is done, before printing the result, the template interpolation is applied, which will result in displaying the content of the namespace r stripping the quotation marks ( -T'{{r}}' -qq )

PS> Disclosure: I'm the creator of the jtc - shell cli tool for JSON operations