Python normalize deeply nested JSON

Question

I am trying to normalize JSON I received from an API via the json_normalize function found in pandas:

{
  "cmd": {
    "success": true,
    "params": {
      "count": 37,
      "result": [
        {
          "id": "5f11c47fb2157c65ba029d4a",
          "orgId": "5d0a54c6b2157c522d409098",
          "name": "tag test",
          "desc": "Removes unnecessary tags",
          "eventType": "campaign.thing",
          "status": "new",
          "ts": "2020-07-17T15:32:15.894Z",
          "summary": {
            "ready": 0,
            "inProgress": 0,
            "success": 0,
            "failure": 0,
            "retry": 0
          },
          "emailUpdates": {},
          "templateGroup": "Tags",
          "templateName": "Tag_Removal",
          "templateId": "5e84f5127094416efc422f67",
          "createdBy": "tester",
          "createdOn": "2020-07-17T15:32:15.894Z"
        },
        {
          "id": "5f11c414b2157c65ba016b35",
          "orgId": "5d0a54c6b2157c522d409098",
          "name": "tag update",
          "eventType": "campaign.thing",
          "status": "new",
          "ts": "2020-07-17T15:30:28.139Z",
          "summary": {
            "ready": 0,
            "inProgress": 0,
            "success": 0,
            "failure": 0,
            "retry": 0
          },
          "emailUpdates": {},
          "templateGroup": "Tags",
          "templateName": "Tag_Add",
          "templateId": "5e84f2fe7094416efc3dd0cd",
          "createdBy": "tester",
          "createdOn": "2020-07-17T15:30:28.139Z"
        }, 
        ...display another 35 JSON objects
      ]
    }
  }
}

Once I receive the response back, I attempt to normalize my code via the python line below:

df_norm = pd.json_normalize(data=Response_JSON, record_path='cmd')

My output is not what I desire, as I end up making a dataframe size (1,3) shown below:

1. cmd.success | cmd.params.count | cmd.params.result
2. True | 37 | [{'id': '5f11c47fb2157c65ba029d4a', 'orgId': '5d0a54c6b2157c522d409098', 'name': ...Rest of JSON above}

The (1,3) cell contains the rest of the JSON in text. The desired output I am looking for would be a columns that drill down further into the JSON. For example, cmd.params.result.id and the containing ids from the JSON object.

It appears that the way my JSON is formatted will not allow it to drill down further. The JSON Normalize Documentation has a meta and record_path parameter, but I've been unsuccessful in getting it to work. Any help would be greatly appreciated!

Answer 1

You can access the nested levels using the following, for your 2 test objects:

df_norm = json_normalize(data=Response_JSON, record_path=['cmd', 'params', 'result'])

...which prints the following with print(df_norm.to_string()) :

                         id                     orgId        name                      desc       eventType status                        ts templateGroup templateName                templateId createdBy                 createdOn  summary.ready  summary.inProgress  summary.success  summary.failure  summary.retry
0  5f11c47fb2157c65ba029d4a  5d0a54c6b2157c522d409098    tag test  Removes unnecessary tags  campaign.thing    new  2020-07-17T15:32:15.894Z          Tags  Tag_Removal  5e84f5127094416efc422f67    tester  2020-07-17T15:32:15.894Z              0                   0                0                0              0
1  5f11c414b2157c65ba016b35  5d0a54c6b2157c522d409098  tag update                       NaN  campaign.thing    new  2020-07-17T15:30:28.139Z          Tags      Tag_Add  5e84f2fe7094416efc3dd0cd    tester  2020-07-17T15:30:28.139Z              0                   0                0                0              0