Python: Convert a JSON structure (including arrays) into a flat, denormalized structure

Question

Given a JSON structure - including array on different levels, like in this example:

{
  "listOfHouses": [
    {
      "name": "House Lannister",
      "listOfMembers": [
        {
          "firstName": "Tywin",
          "lastName": "Lannister"
        },
        {
          "firstName": "Cersei",
          "lastName": "Lannister"
        }
      ]
    },
    {
      "name": "House Targaryen",
      "listOfMembers": [
        {
          "firstName": "Daenerys",
          "lastName": "Targaryen"
        }
      ]
    }
  ]
}

I tried several approaches (eg using pd.json_normalize(...) or recursive tree algorithmus), but none of them returns result where each and every member of a house has its OWN entity in a converter structure. I want the following converted structure:

{
  {
    'name': 'House Lannister',
    'firstName': 'Tywin',
    'lastName': 'Lannister'
  }, {
    'name': 'House Lannister',
    'firstName': 'Cersei',
    'lastName': 'Lannister'
  }, {
    'name': 'House Targaryen',
    'firstName': 'Daenerys',
    'lastName': 'Targaryen'
  }
}

Can this be implemented in Python in a generic way?

Tried the following 2 approaches, both did not work to my satisfaction:

import pandas as pd
import json

def flatten_json(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

if __name__ == '__main__':
    with open('got-houses.json') as json_file:
        data = json.load(json_file)
        normalized_json = pd.json_normalize(data, 'listOfHouses')
        print(normalized_json.to_string())
        
        flattened_json = flatten_json(data['listOfHouses'])
        print(flattened_json)

Output for pd.json_normalize:

0  House Lannister  [{'firstName': 'Tywin', 'lastName': 'Lannister'}, {'firstName': 'Cersei', 'lastName': 'Lannister'}]
1  House Targaryen                                                 [{'firstName': 'Daenerys', 'lastName': 'Targaryen'}]

Output for tree-walk:

{
  '0_name': 'House Lannister',
  '0_listOfMembers_0_firstName': 'Tywin',
  '0_listOfMembers_0_lastName': 'Lannister',
  '0_listOfMembers_1_firstName': 'Cersei',
  '0_listOfMembers_1_lastName': 'Lannister',
  '1_name': 'House Targaryen',
  '1_listOfMembers_0_firstName': 'Daenerys',
  '1_listOfMembers_0_lastName': 'Targaryen'
}

Answer 1

I believe the below is what you are looking for. (just a nested loop)

data = {
    "listOfHouses": [
        {
            "name": "House Lannister",
            "listOfMembers": [
                {
                    "firstName": "Tywin",
                    "lastName": "Lannister"
                },
                {
                    "firstName": "Cersei",
                    "lastName": "Lannister"
                }
            ]
        },
        {
            "name": "House Targaryen",
            "listOfMembers": [
                {
                    "firstName": "Daenerys",
                    "lastName": "Targaryen"
                }
            ]
        }
    ]
}
output = []
for entry in data['listOfHouses']:
    for sub in entry['listOfMembers']:
        output.append({'name': entry['name'], 'firstName': sub['firstName'], 'lastName': sub['lastName']})
print(output)

output

[{'name': 'House Lannister', 'firstName': 'Tywin', 'lastName': 'Lannister'}, {'name': 'House Lannister', 'firstName': 'Cersei', 'lastName': 'Lannister'}, {'name': 'House Targaryen', 'firstName': 'Daenerys', 'lastName': 'Targaryen'}]