Parse complex JSON in Python

Question

EDITED WITH LARGER JSON:

I have the following JSON and I need to get id element: 624ff9f71d847202039ec220

results": [
    {
      "id": "62503d2800c0d0004ee4636e",
      "name": "2214524",
      "settings": {
        "dataFetch": "static",
        "dataEntities": {
          "variables": [
            {
              "id": "624ffa191d84720202e2ed4a",
              "name": "temp1",
              "device": {
                "id": "624ff9f71d847202039ec220",
                "name": "282c0240ea4c",
                "label": "282c0240ea4c",
                "createdAt": "2022-04-08T09:01:43.547702Z"
              },
              "chartType": "line",
              "aggregationMethod": "last_value"
            },
            {
              "id": "62540816330443111016e38b",
              "device": {
                "id": "624ff9f71d847202039ec220",
                "name": "282c0240ea4c",
              },
              "chartType": "line",
            }
          ]
        }
    ...

Here is my code (EDITED)

    url = "API_URL"
    response = urllib.urlopen(url)
    data = json.loads(response.read().decode("utf-8"))
    print url
    all_ids = []
    for i in data['results']:  # i is a dictionary
       for variable in i['settings']['dataEntities']['variables']:
           print(variable['id'])
           all_ids.append(variable['id'])

But I have the following error:

        for variable in i['settings']['dataEntities']['variables']:
KeyError: 'dataEntities'

Could you please help? Thanks!!

Answer 1

What is it printing when you print(fetc) ? If you format the json, it will be easier to read, the current nesting is very hard to comprehend.

fetc is a string, not a dict. If you want the dict, you have to use the key.

Try:

url = "API_URL"
response = urllib.urlopen(url)
data = json.loads(response.read().decode("utf-8"))
print url
for i in data['results']:
    print(json.dumps(i['settings']))
    print(i['settings']['dataEntities']

EDIT: To get to the id field, you'll need to dive further.

i['settings']['dataEntities']['variables'][0]['id']

So if you want all the ids you'll have to loop over the variables (assuming the list is more than one)`, and if you want them for all the settings, you'll need to loop over that too.

Full solution for you to try (EDITED after you uploaded the full JSON):

url = "API_URL"
response = urllib.urlopen(url)
data = json.loads(response.read().decode("utf-8"))
print url

all_ids = []
for i in data['results']:  # i is a dictionary
    for variable in i['settings']['dataEntities']['variables']:
        print(variable['id'])
        all_ids.append(variable['id'])
        all_ids.append(variable['device']['id']

Let me know if that works.

Answer 2

The shared JSON is not valid. A valid JSON similar to yours is:

{
  "results": [
    {
      "settings": {
        "dataFetch": "static",
        "dataEntities": {
          "variables": [
            {
              "id": "624ffa191d84720202e2ed4a",
              "name": "temp1",
              "span": "inherit",
              "color": "#2ccce4",
              "device": {
                "id": "624ff9f71d847202039ec220"
              }
            }
          ]
        }
      }
    }
  ]
}

In order to get a list of ids from your JSON you need a double for cycle. A Pythonic code to do that is:

all_ids = [y["device"]["id"] for x in my_json["results"] for y in x["settings"]["dataEntities"]["variables"]]

Where my_json is your initial JSON.