Using Python module Glom, Extract Irregular Nested Lists into a Flattened List of Dictionaries

Question

Glom makes accessing complex nested data structures easier. https://github.com/mahmoud/glom

Given the following toy data structure:

target = [
            {
                'user_id': 198,
                'id': 504508,
                'first_name': 'John',
                'last_name': 'Doe',
                'active': True,
                'email_address': 'jd@test.com',
                'new_orders': False,
                'addresses': [
                    {
                        'location': 'home',
                        'address': 300,
                        'street': 'Fulton Rd.'
                    }
                ]
            },
            {
                'user_id': 209,
                'id': 504508,
                'first_name': 'Jane',
                'last_name': 'Doe',
                'active': True,
                'email_address': 'jd@test.com',
                'new_orders': True,
                'addresses': [
                    {
                        'location': 'home',
                        'address': 251,
                        'street': 'Maverick Dr.'
                    },
                    {
                        'location': 'work',
                        'address': 4532,
                        'street':  'Fulton Cir.'
                    },
                ]
            },
        ]

I am attempting to extract all address fields in the data structure into a flattened list of dictionaries.

from glom import glom as glom
from glom import Coalesce
import pprint

"""
Purpose: Test the use of Glom
"""    

# Create Glomspec
spec = [{'address': ('addresses', 'address') }]

# Glom the data
result = glom(target, spec)

# Display
pprint.pprint(result)

The above spec provides:

[
    {'address': [300]},
    {'address': [251]}
]

The desired result is:

[
    {'address':300},
    {'address':251},
    {'address':4532}
]

What Glomspec will generate the desired result?

Answer 1

As of glom 19.1.0 you can use the Flatten() spec to succinctly get the results you want:

from glom import glom, Flatten

glom(target,  (['addresses'], Flatten(),  [{'address': 'address'}]))
# [{'address': 300}, {'address': 251}, {'address': 4532}]

And that's all there is to it!

You may also want to check out the convenient flatten() function , as well as the powerful Fold() spec , for all your flattening needs :)

---

Prior to 19.1.0, glom did not have first-class flattening or reduction (as in map-reduce) capabilities. But one workaround would have been to use Python's built-in sum() function to flatten the addresses:

>>> from glom import glom, T, Call  # pre-19.1.0 solution
>>> glom(target,  ([('addresses', [T])], Call(sum, args=(T, [])),  [{'address': 'address'}]))
[{'address': 300}, {'address': 251}, {'address': 4532}]

Three steps:

1. Traverse the lists, as you had done.
2. Call sum on the resulting list, flattening/reducing it.
3. Filter down the items in the resulting list to only contain the 'address' key.

Note the usage of T , which represents the current target, sort of like a cursor.

Anyways, no need to do that anymore, in part due to this answer. So, thanks for the great question!